Noise Consistency Training: A Native Approach for One-step Generator in Learning Additional Controls

We sincerely thank you for your time and effort in reviewing our paper. We address each comment below.

The overly simplistic reduction of ControlNet's architecture to a single parameter does not accurately reflect its structure/contributions.

Our NCT is agnostic to the specific structure of the ControlNet or adapter. The notation $\phi$ is used as a general representation of the adapter's learnable parameters, not as a reduction of its complexity. Whether the adapter is a standard ControlNet, a lightweight adapter like IP-Adapter, or any other control module, our training methodology applies uniformly. Our work is not in modifying ControlNet's architecture, but in developing an effective training strategy for any control adapter when applied to one-step generators. We chose this general notation precisely because our method works across different adapter architectures without requiring architecture-specific modifications.

Figure 1 lacks discussion or analysis in main text. Its relevance to NCT's mechanism or results is unclear.

Figure 1 is the overall framework description of NCT with a clear caption and contents. We will provide a brief introduction of Figure 1 in the method section to make it clearer.

The chaining of three expectation operators is unnecessarily complex and hinders readability.

We have simplified the notation in the second line of Eq. (6).

The presence of #Simplify Notation is unacceptable and suggests unrevised drafting artifacts.

The "#Simplify Notation" is to clearly indicate that we are simplifying notation in Line 2 from Line 1 of Eq. (6).

The subscript $z,c|z,\epsilon$ appears erroneous and logically inconsistent (z appears conditioned on itself?).

This denotes three variables rather than one variable, where each is separated by ",".

The symbol ($|| \cdot ||_1$) lacks specification (e.g., L1, L2).

We note that the symbol $||\cdot||_1$ exactly denotes the L1 norm.

The core innovation, NCT, appears heavily reliant on applying a consistency loss principle (well-established in consistency models) to the specific context of adapting ControlNet-like structures for one-step generators. While the combination is novel and practically useful, the paper fails to sufficiently differentiate the fundamental contribution beyond this synergistic application.

NCT is not a combination of Consistency Training (CT) and ControlNet. There is a fundamental and significant difference between Noise Consistency Training (NCT) and CT. NCT "diffuses" noise in the noise space and "denoises" the noise to learn adapters for adding controls to one-step generators, while CT adds noise to data in the data space and denoises the noisy data to learn few-step generator.

The discussion lacks depth in contrasting NCT with relevant alternative approaches for efficient adaptation or conditional generation (Shortcut Models and Meanflows).

NCT is orthogonal to the specific method used for learning the one-step generator. Since NCT focuses on injecting conditions into a well-trained, one-step generator, which is agnostic to the method used to train the one-step generator. Furthermore, current approaches like Shortcut Models and Meanflows do not prioritize controllable generation, nor have they demonstrated the capability for high-quality, one-step text-to-image synthesis. In contrast, NCT is capable of achieving high-quality, controllable text-to-image generation in one step.

Lemmas&theorems strongly resemble those from the foundational consistency model literature without clear significant adaptation specific to NCT's conditional control objective. The paper does not convincingly establish what new theoretical insight NCT provides.

We respectfully argue that NCT is not a derivative work but a novel approach with a fundamental distinct theoretical and algorithmic foundation.

The core difference is that NCT enforces consistency in the noise space, while Consistency Models (CMs) operate in the data space. These Lemmas&theorems are necessary reformulation, not a minor adaptation. In particular, NCT "diffuses" noise in the noise space and "denoises" the noise, while CMs adds noise to data in the data space and denoises the noisy data. NCT is designed for pre-trained one-step generators to add controls, which lack the multi-step PF-ODE trajectory originating from a clean data point that CMs fundamentally rely on.

This algorithmic shift necessitates a different theoretical underpinning. While CM theory is rooted in approximating ODE solutions, NCT's theory is based on distribution matching. Our noise diffusion process creates an interpolation path from a coupled to an independent latent-condition distribution. The NCT objective, including the crucial boundary loss, forces the model's output distribution to remain invariant along this path.

There's a disconnect between the listed lemmas/theorems and the core arguments or practical results of the paper. A stronger narrative linking theory to the proposed method and its empirical success is missing.

We will revise the manuscript to make the connection between our theory and practice more explicit. The narrative is as follows:

Our main theorem establishes the ideal objective. It proves that a generator can learn the true conditional posterior if it satisfies two abstract conditions: a Boundary Condition and a Consistency Condition.

Our proposed method is a direct, practical implementation of this theory. The theoretical Boundary Condition is directly optimized by our Boundary Loss ($L_{bound}$). The theoretical Consistency Condition (formalized as an MMD objective in Lemma is empirically approximated by our Noise Consistency Loss ($L_{con}$).

The strong empirical results are the validation of this framework. They demonstrate that our practical losses effectively serve as proxies for the theoretical ideals, successfully guiding the model to the desired conditional distribution. Our ablation study, which shows complete model failure when either loss is removed, further reinforces that both theoretical pillars (and their practical counterparts) are essential.

Attributing blurry images solely to "high variance of the optimized objective" is asserted without analysis, proof, or citation. What variance? Compared to what?

The "high variance" refers to the variance in the conditional distribution $p(x|x_T)$ when optimizing the direct denoising objective. When $x_T$ is pure noise, there are infinitely many possible clean images $x$ that could have generated it, making $p(x|x_T)$ highly multimodal. The optimal solution for minmizing the direct denoising loss under an L2 loss is $\mathbb{E}[x|x_T]$, which averages over all possible images in the dataset, inherently producing blurry results.

To substantiate this claim, we conducted an empirical analysis measuring the variance of the optimization objectives:

The results clearly demonstrate that the noise consistency loss exhibits significantly lower variance compared to direct denoising, supporting our assertion.

The claim about the model's inability to perform conditional generation given random z due to training on coupled pairs (z, c) is stated dogmatically. No theoretical argument or experimental evidence is provided.

We have ablated this variant in our main paper, showing that it is unable to handle the condition. In particular, training on coupled pairs (z, c) is equal to our ablated variant "w/o noise consistency training". Results are in Figure 3 and Table 4, indicating that this approach is unable to inject conditions.

Limited Resolution

Our experiments are conducted over 512 resolution, which is already relatively high-resolution and widely adopted in most prior works in the line of distillation [3, 5, 9] and also the original multi-step ControlNet [12].

Narrow Condition Scope.

Our experiments focused on controllable generation (based on ControlNet) and Image-prompted generation (IP-Adapter). We believe this demonstrates NCT's broad applicability rather than limiting it.

Our experimental design deliberately selected these two architectures because they represent fundamentally different conditioning paradigms: ControlNet exemplifies spatial/structural control (edges, depth, HED, low-resolution) while IP-Adapter represents semantic/style control through image prompts. Together, they span the spectrum from low-level geometric constraints to high-level semantic guidance.

Moreover, ControlNet and IP-Adapter serve as foundational architectures for numerous downstream applications. We believe that NCT's experiments have covered a sufficient number of conditional scenarios.

How essential is the specific adapter architecture?

Our NCT is agonistic to specific adapter architecture, since NCT is an algorithm in training adapters to add new controls for one-step generators. We apply the same adapter architecture among methods in our experiments. NCT can effectively train adapters for one-step generator in adding new controls, as indicated by the success on both ControlNet and IP-Adapter architecture.

What is the impact of the noise consistency loss vs. simpler losses?

We have ablated that simply use boundary loss without noise consistency loss in Table 3 and Figure 4. It can be seen that without noise consistency loss, it is unable to inject conditions. Additionally, we conduct an extra ablation study on simply using direct denoising loss:

We sincerely thank you for your time and effort in reviewing our paper. We address each comment below.

The paper can be difficult to follow for readers who are not already familiar with the consistency models. For instance, the presentation of consistency models and Equation (4), which defines the consistency loss, lacks clarity.

Thank you for your valuable feedback. We will expand the preliminary section to include a comprehensive introduction to consistency models, ensuring readers unfamiliar with this framework can follow our work more easily. Besides, while consistency models serve as inspiration for our noise consistency training, we want to clarify that our approach is fundamentally self-contained. The key differences include: 1) our noise consistency training operates in a different space and uses a distinct formulation; 2) the theoretical foundations and practical implementation differ significantly from standard consistency models.

Could the authors provide a clearer explanation of why consistency training can force inject conditioning signals, and definition of the stop-gradient operator?

For consistency training with condition $c$, it tries to make $g(x_{t_{n+1}}, c)$ approximate $g(x_{t_{n}}, c)$, where $t_{n+1} > t_n$ and $g$ is the desired consistency model, taking noisy samples with condition and predicts clean samples. Intuitively, the network can better predict clean samples given less noisy samples $x_{t_{n}}$, hence the $g(x_{t_{n}}, c)$ can be used as the target for learning $g(x_{t_{n+1}}, c)$. In this case, the information in $x_{t_{n+1}}$ about the image is more corrupted than $x_{t_{n}}$, which can force the network to utilize the information from condition $c$ to minimize the loss.

The stop-gradient operator treats a variable as a constant, preventing gradients from backpropagating through it.

Moreover, the paper suffers from inconsistent and overloaded notation timesteps $t_n, t_k$, are used ambiguously as subscripts for both $x$ and $z$.

Sorry for the confusion. We will clean up the notation in our revision.

The definitions of diffusion parameters in later equations (e.g., (5), (6)) are inconsistent with those used earlier.

We deliberately use different symbols to distinguish between different models. Specifically, Eq. 4 is a standard consistency model ($g_\alpha$), while Eq. 6 involves a one-step generator ($f_{\theta}$) and its adapted version ($f_{\theta,\phi}$).

The definition of $p(c|x)$

We model the distribution $p(c|x)$ as a Dirac delta distribution: $p(c|x) \triangleq \delta(c - h(x))$, where $h(\cdot)$ represents a deterministic discriminative model that maps inputs to conditions. The Dirac delta distribution can be sampled easily by $h(x)$, but it is hard to estimate the density. This is well-suited for our NCT, since NCT just requires samples from $p(c|x)$ for training.

Lemma 1 makes its claim without stating the necessary assumptions in the main text. While the appendix provides a specific parameter scheduling under which the result holds this renders the lemma rather trivial. However, the main paper does not clearly state this parameter scheduling.

We are sorry for the confusion on the scheduler. We note that our schedule is specified by $z_t = \sigma_t z + \sqrt{1-\sigma_t}\epsilon$. And we acknowledge a typo in Eq. 5, where the $z_T=\sigma_t z + \sqrt{1-\sigma_t}\epsilon$ should be $z_t=\sigma_t z + \sqrt{1-\sigma_t}\epsilon$. Besides, $z_T$ denotes the terminal diffused noise (i.e., $\sigma$ = 1), while $z_0$ denotes the initial noise (i.e., $\sigma$ = 0). This is a widely used notation in diffusion literature, and we will clarify this in the revision.

Regarding Lemma 1, it is given for formally stating that our diffusion process can obtain independent noise-condition pairs, which is crucial for enabling our approach to learn conditional generation. Although the approach is straightforward, it provides a formal ground for subsequent analysis.

The paper repeatedly asserts that the proposed noise consistency loss yields lower variance, yet it offers neither theoretical justification nor empirical evidence to support this claim. I believe this is true, but could the authors give a more precise explanation?

The lower variance is compared to the direct denoising objective. For the direct denoising objective, the target samples follow conditional distribution $p(x|x_T)$ which is highly multimodal. Since when $x_T$ is pure noise, there are infinitely many possible clean images $x$ that could have generated it. In contrast, for noise consistency loss, the target samples follows $q(x|z_{t_{n}},c) = \int p(z_{t_{n}}|z_{t_{n+1}})p(x|z_{t_n},c)$, where $p(x|z_{t_n},c)\triangleq \delta(x-f(z_{t_n},c))$. And $p(z_{t_{n}}|z_{t_{n+1}})$ has small variance, since the timestep gap between $z_{t_{n+1}}$ and $z_{t_{n}}$ is chosen to be small. Based on the above intuition, we believe our method produces lower variance.

To substantiate this claim, we conducted an empirical analysis measuring the variance of the optimization objectives:

The results clearly demonstrate that the noise consistency loss exhibits significantly lower variance compared to direct denoising, supporting our assertion.

We sincerely thank you for your time and effort in reviewing our paper. We address each comment below.

Insufficient Ablation Study : In Section 3.1, the paper claims that a direct denoising approach yields "blurry images," and while Table 3 shows a very high FID for the "w/o noise consistency loss" ablation, it fails to provide a visual example, unlike the other ablations in Figure 4. This makes the claim less convincing, as it is difficult to assess the degree and nature of the "blurriness" based solely on the FID score.

The results for the 'w/o noise consistency' variant are presented in both Figure 4 and Table 3. This variant is distinct from direct denoising, for which we report the results below:

It can be seen that direct denoising shows much worse FID due to its blurry samples. For visual samples, we will provide them in the revision. We cannot provide it in the rebuttal stage due to the policy. However, we refer to Figure 3 of JDM, which shows results of direct denosing with an additional distillation loss (Diff-instruct loss). It can be seen that the direct denoising approach still produces blurry samples even with an additional distillation loss.

It is worth noting that the reason why the variant (w/o noise consistency loss) exhibits a worse FID score compared to NCT is that the FID calculation is performed between the real images and the samples generated under the conditions corresponding to the same real images. Controllable generation with $c\sim p(c|x)$ allows for a distribution closer to that of the real image $x$.

The paper compares against IP-Adapter in Table 2 but not in Table 1

This is because the IP-Adapter in Table 2 and ControlNet in Table 1 focus on different tasks. ControlNet is concerned with controllable generation, aiming to preserve the details of the given conditions. In contrast, the IP-Adapter focuses on image-prompted generation, where the main goal is to inject the semantics and style of a reference image. This makes it unsuitable to evaluate them using the same benchmark.

Compare JDM regarding image-prompted generation

Great suggestion. We compare JDM regarding image-prompted generation below:

It can be seen that NCT achieves competitive results compared to JDM, which requires additional expensive distillation loss.

On a Fairer Comparison with ControlNet: The proposed NCT methodology appears to be applicable not only to one-step generators but also to standard multi-step diffusion models. To provide a more direct and fair comparison, could you apply NCT on a standard diffusion model (e.g., Stable Diffusion 1.5) and benchmark it against the original ControlNet using the exact same base model and evaluation settings (e.g., 50 NFEs)? The current comparisons in Table 1 are challenging to interpret on a one-to-one basis, as they compare models with different base architectures and step counts. This new experiment would help isolate the performance of the training methodology itself.

We first clarify that the network architecture (Controlnet + UNet) adopted by NCT is identical to that of standard ControlNet. The difference is that NCT trains a ControlNet for a one-step generator distilled from a pre-trained diffusion model, whereas standard ControlNet trains a ControlNet for the diffusion model itself. NCT actually faces a greater challenge because it needs to inject the condition in one-step generation. Hence, we believe the comparison is fair for baselines and NCT's advantage is convincing.

NCT is tailored for one-step generators, and extending it to standard diffusion models presents some difficulties. The reason is: NCT training requires access to the model's predicted $x_0$ and its corresponding condition $c$. This is not feasible for diffusion models because their one-step predictions are blurry, and $p(c|x)$ is trained on clean images, making it difficult to obtain the accurate corresponding conditions needed for training NCT. Exploring the extension of NCT to standard diffusion models would be interesting future work.

Could you please provide side-by-side visual comparisons of your method's outputs against those from other one-step controllable generation models, such as JDM?

Thanks for the great suggestion. We will provide the visual samples in revision. But limited to the policy of rebuttal, we cannot provide the samples in the rebuttal stage. Sorry for this.

On the Generalizability of NCT: The experiments currently focus on structural control signals, such as Canny edges and depth maps. Have you explored the efficacy of NCT with non-structural or stylistic control signals, for instance, controlling for a specific texture or an artistic style? Could you elaborate on how the core assumption of NCT—that diffusing noise progressively decouples it from the condition—is expected to hold for features that are less spatially robust? For such controls, even minor perturbations in the noise space might lead to significant semantic deviations, potentially challenging the consistency objective.

We highlight that, theoretically, using NCT to inject conditions for one-step generation does not depend on what the condition is. In particular, in Theorem 1, we show that our optimization objective can learn a conditional generator $f_{\theta,\phi}(x,c)$, and that the validity of this theorem is independent of the type of the condition $c$.

Empirically, for the generation of injected image styles, we already have a visualized image based on IP-Adapter in our main paper (Figure 3). In the second row of Figure 3, we can effectively use Van Gogh's paintings as an image prompt to inject style, which can effectively infuse the model-generated images with the Van Gogh style.

Thank you for your reply. I have some more questions.

1. Thank you for comparing your results to JDM. I think even if it scores similar to JDM your methods is more elegant, efficient. I think NCT has advantages than JDM. I hope authors should include the table compared to JDM and NCT, with qualitative comparison later.

2. Regarding Style-Controlled Generation

For my understanding, the input is (x,c) and the author mentioned that "we can effectively use Van Gogh's paintings as an image prompt to inject style", however, It does not mean inject style as controllable condition. It just processed images as input. So, the questions about high-frequency feature, are not addressed. For example, you can condition magnitude of images as condition after DFFT, as because magnitude are consider to be style. What I am curious is that, For my understanding, you intuition is built upon "if you inference similar noise, you will got similar images", because it is one-step generation.

Since most of my concerns are solved, I am willing to adjust my score in final decision stage.

We sincerely thank you for your time and effort in reviewing our paper. We address each comment below.

It is not clearly explained the notion of $p(c|x)$. Do we assume we have online access to turning samples into the underlying condition?

The $p(c|x)$ is indeed for sampling the underlying condition $c$ regarding $x$. This is for constructing the triplet $(\epsilon,x,c)$ for conducting NCT's training, where the diffusing process in noise space is performed on $(\epsilon,c)$ and the boundary is defined by $(\epsilon,x)$.

What if the control signal is data-driven? Specifically, if we assume that we have access to paired data of conditions and samples but no model outputting conditions given samples, we could still do ControlNet then distillation, but the proposed method here will not work, correct?

Thanks for the great question. We note that the training of NCT requires triplets $(\epsilon,x,c)$. Hence, our method can be extended to the data-driven case by a minor modification: train an encoder to invert $x$ to noise $\epsilon$, then formulate the triplet $(\epsilon,x,c)$ for performing NCT's training. We note that training the encoder may be expensive and difficult for diffusion models, but it is easy for a one-step generator. In particular, the encoder can be trained by minimizing reconstruction loss: $\mathrm{Min} _ { \mathrm{Enc} \in \mathcal{F}} | | f_\theta(\mathrm{Enc}(x)) - x | | _2^2$. We train an encoder initialized from DMs in just 5,000 iterations. We report NCT trained by data-driven control signal as follows:

It can be seen that the data-driven NCT can also achieve promising results.

I suggest adding something indicating noise injection in Figure 1. Its current version is not very intuitive to understand, and all 4 noises look exactly the same while there are 3 levels.

Great suggestion. We will improve the Figure 1 in the revision. Specifically, we will add some structural/semantic features to the noise to enhance readability, rather than faithfully rendering standard Gaussian noise.

Is the primal-dual algorithm really necessary? Have you tried sweeping different values of the dual weight? Related question: What is the computational overhead of the primal-dual algorithm, compared to just primal ( $\lambda$ is a hyperparameter)?

Without primal-dual can also work reasonably well as shown in Table 3. The major advantage of primal-dual is that it can enable the dynamic tuning of $\lambda$. Notably, it can allow $\lambda$ to be zero, when the boundary is nearly satisfied. This can not be achieved by any fixed $\lambda$ value (except zero).

The computational overhead is negligible, since we just need to compute $\lambda_{t+1} = \mathrm{max}[{\lambda_t + \eta (L_{con} - \xi),0}]$ without additional network forward and backward, which is highly efficient.