Discrete Modeling via Boundary Conditional Diffusion Processes

We sincerely appreciate your meticulous review and the valuable comments. Please kindly find our response below.

Q1: The rescaled vector field and probability path

Due to the character limit, this is only an outline. Please refer to comments for more details.

1. Our neural network predicts $\mathbf{x}_0$ not $\tilde{u}_t(\tilde{\mathbf{x}}_t|\mathbf{x}_0)$.

2. $\tilde{u}_t(\tilde{\mathbf{x}}_t|\mathbf{x}_0)$ is tractable but we did not derive it because it is not used.

3. We generate the probability path with the deterministic reverse diffusion process (Equation 25).

4. The training objective $||\mathbf{x}_0-\mathbf{x} _\theta||$ is valid. Derivations in Comments.

Q2: Method and the probability contour

Due to the character limit, this is only an outline. Please refer to comments for more details.

1. We have demonstrated the rescaled probability contours of $\tilde{p}_t(\tilde{\mathbf{x}}_t|\mathbf{x}_0)$ in Figure 2, where sample points are exactly calculated by our proposed method (Equation 22).

2. $\tilde{p}_t(\mathbf{x}|\mathbf{x}_0)$ is a series of functions, where the consistency to the boundary contour is inversely proportional to the subscript $t$.

Q3: Confidence factor

Due to the character limit, this is only an outline. Please refer to comments for more details.

1. The confidence factor is named after some thoughts of sampling, such as the confidence interval. Our data is discrete and the uncertainty comes from the learned diffusion model.

2. The mode collapse when $r=1$ comes from two part: One is the uncertainty of the learned diffusion model as described above. The other involves a traditional problem of diffusion processes.

3. Fixing the initial path means the Trajectory Alteration (Line 8 in Algorithm2) is optional in practice. This is a negligible detail of our model.

Q4: Re-ranking

This is a fair comparision because beam searching the generated results of the transformer is re-ranking with transformer.

Re-ranking is a traditional strategy widely used in Non-autoregressive translation[7,8]. We use the generated results of our method re-ranked by the transformer to demonstrate their upper bound. Since the generation of diffusion LMs are highly random, their own probability scores often cannot reflect the upperbound of their performance. Re-ranking the generated results with an auto-regressive model can better demonstrate the potential capability of the diffusion model. Besides, re-ranking the transformer baseline just gets what it generates. Because beam search is exactly re-ranking the generated contents with the transformer's probabilities themselves.

Q5: Table 3

1. We believe it is a reasonable comparision because we strictly follows the setting of BitDiffusion[9] as in the Table 1 of their paper. We want to clarify that the embedding strategy is only valid for continuous diffusion models, where there is not embedding stage for discrete diffusion models. We compare the continuous diffusion models and the discrete diffusion models under the same level of continuous information, Continuous Pixels, Discrete Ordinal Pixels, and Categorical Pixels, where the latter the less continuous information models can obtain.

2. We have clarified in Lines 285-287 that since they only provide the model code without a training script with detailed hyperparameters or a checkpoint, we have to reproduce with exactly the same configuration based on their code and paper. We have done our best effort to reproduce the results and we promise to release our code and reproduction. Besides, we achieve the best reproduction result in the open source community. We are confident in the reliability of our experimental results and hope to address the reviewer's concerns.

3. Even ignoring the above issues, our method still shows effectiveness. Considering the gap introduced by the continuous information that the original BitDiffusion (Fid 3.48) can not beat DDPM (Fid 3.17) under the Gaussian sampling, our method (Fid 3.86) achieves improvement over DDIM (FiD 4.04), which are both deterministic diffusion processes. This means that our approach can achieve better results with less continuous information.

Q6: Samples in the boundary

1. It is possible that a random noise $\boldsymbol{\epsilon}$ is sampled within $\mathbf{x}_0$'s discrete boundary. However, the probability of this happening in a high-dimensional representation space is very low. According to our statistics during the training phase, this probability is less than 0.1%.

2. Our solution is demonstrated in the pseudo code (Line 639, Appendix F), where we mask out this situation when $f(\boldsymbol{\epsilon},\mathcal{I})>f(\boldsymbol{\epsilon},\mathcal{J})$. This means that random noise $\boldsymbol{\epsilon}$ sampled within $\mathbf{x}_0$'s discrete boundary will not be rescaled and its trajectory is the same as original DDPM.

Q7: Line 187

Similar to 3.2 in Q1, as demonstrated in Lines 186-189, we provide an alternative for the ODE decoding, which is a deterministic reverse process. The word 'deterministic' is used to reflect the difference between our approach and traditional Gaussian process in DDPM, because both of them model the reverse process.

Q8: Ablation in Table 2

1. First three lines in Table 2 reveal where the performance improvement of our method mainly comes from. Only rescale the forward process means we just train the diffusion model with the rescaled trajectory and reverse the trajectory as in the DDPM. Rescaling the forward and backward means we use the rescaled trajectory for both training and inference. The results in Table 2 shows that training with the rescaled trajectories makes a major contribution to the final performance of our method.

2. Last line in Table 2 reveals that our method is robust for different trajectory types that optimal transport trajectory used in Flow Matching can sometimes achieve better results.

Q1: The rescaled vector field and probability path

We do not define the vector field $\tilde{u}_t(\tilde{\mathbf{x}}_t|x_0)$ in our framework, because our neural network predicts $\mathbf{x}_0$ and we generate the probability path with the deterministic reverse diffusion process.

1. Why do we predict $\mathbf{x}_0$?

$G(\mathbf{x}_0,\boldsymbol{\epsilon})$ is the key of our method during both training and inference stage, we are required to have a low error estimation on $\mathbf{x}_0$ (Lines 180-182). Predicting $\tilde{u}_t(\tilde{\mathbf{x}}_t|x_0)$ or $\boldsymbol{\epsilon}$ will amplify the errors in the neural network's output.

2. How to define the vector field $\tilde{u}_t(\tilde{\mathbf{x}}_t|\mathbf{x}_0)$ if we want?

Preliminary: $u_t(\mathbf{x}_t|\mathbf{x}_0) = \frac{\mathrm{d}\mathbf{x}_t}{\mathrm{d}t}$ (Equation 11-13 in Flow Matching [1])

In our framework: $\tilde{\mathbf{x}}_t = \mathbf{u}(\mathbf{x}_0,\mathcal{T}(t,G(\mathbf{x}_0,\boldsymbol{\epsilon})))\mathbf{x}_0+\mathbf{v}(\mathbf{x}_0,\mathcal{T}(t,G(\mathbf{x}_0,\boldsymbol{\epsilon})))\boldsymbol{\epsilon}$ (Equation 22)

Therefore: $\tilde{u}_t(\tilde{\mathbf{x}}_t|\mathbf{x}_0) = \frac{\mathrm{d}\tilde{\mathbf{x}}_t}{\mathrm{d}t} = \left[\mathbf{u}'(\mathbf{x}_0,\mathcal{T}(t,G(\mathbf{x}_0,\boldsymbol{\epsilon})))\mathbf{x}_0+ \mathbf{v}'(\mathbf{x}_0,\mathcal{T}(t,G(\mathbf{x}_0,\boldsymbol{\epsilon})))\boldsymbol{\epsilon}\right]\frac{\mathrm{d}\mathcal{T}(t,G(\mathbf{x}_0,\boldsymbol{\epsilon}))}{\mathrm{d}t}$

Given: $\mathcal{T}(t,t_0) = r\times t_0 + t\times (T-r\times t_0)/T$ (Eqaution 19), we have $\frac{\mathrm{d}\mathcal{T}(t,G(\mathbf{x}_0,\boldsymbol{\epsilon}))}{\mathrm{d}t}=\frac{T-r\times G(\mathbf{x}_0,\boldsymbol{\epsilon})}{T}$

Let $\tau$ denote $\mathcal{T}(t,G(\mathbf{x}_0,\boldsymbol{\epsilon}))$

Hence: $\tilde{u}_t(\tilde{\mathbf{x}}_t|\mathbf{x}_0) = \frac{\mathrm{d}\tilde{\mathbf{x}}_t}{\mathrm{d}t} = \left[\mathbf{u}'(\mathbf{x}_0,\tau)\mathbf{x}_0+ \mathbf{v}'(\mathbf{x}_0,\tau)\boldsymbol{\epsilon}\right]\frac{T-r\times G(\mathbf{x}_0,\boldsymbol{\epsilon})}{T}$, which is actually tractable.

3. How to generate the probability paths?

• 3.1 One direct solution is Equation 24, which is derived from the Theorem3 in Flow Matching[1]

Let $\tilde{p}_t(\mathbf{x}|\mathbf{x}_0)$ be a probability path, given equation 22, there is $\boldsymbol{\epsilon}=\frac{\mathbf{x}-\mathbf{u}(\mathbf{x}_0,\tau)\mathbf{x}_0}{\mathbf{v}(\mathbf{x}_0,\tau)}$.

Replacing $\boldsymbol{\epsilon}$ in $\tilde{u}_t(\tilde{\mathbf{x}}_t|\mathbf{x}_0)$ with $\mathbf{x}$, we have $\tilde{u}_t(\mathbf{x}|\mathbf{x}_0)= \left[ \frac{\mathbf{v}'(\mathbf{x}_0,\tau)}{\mathbf{v}(\mathbf{x}_0,\tau)}(\mathbf{x}-\mathbf{u}(\mathbf{x}_0,\tau)\mathbf{x}_0)+\mathbf{u}'(\mathbf{x}_0,\tau)\mathbf{x}_0\right]\frac{T-r\times G(\mathbf{x}_0,\boldsymbol{\epsilon})}{T}$.

Therefore, we generate the probability path with $\tilde{u}_t(\mathbf{x}|\mathbf{x}_0)$.

However, this may be inefficient in practical use because we have to solve the equation $\tau=\mathcal{T}(t,G(\mathbf{x}_0,\frac{\mathbf{x}-\mathbf{u}(\mathbf{x}_0,\tau)\mathbf{x}_0}{\mathbf{v}(\mathbf{x}_0,\tau)}))$ to get the $\tau$ with respect to the change of $\mathbf{x}$ in real time. This is tractable for special cases such as original Flow Matching (Equation 42 in Appendix E), but it will be much more complex for Diffusion trajectories (Equation 43 in Appendix E). Since our approach is a general framework for all continuous diffusion processes, including both Diffusion Model and Flow Matching, we actually use an alternative approach similar to the DDIM[2].

It's worth noting that we just demonstrate the vector field function $u_t(\mathbf{x}|\mathbf{x}_0)$ in Equation 24 to show its complexity. We do not provide the derivation of $\tilde{u}_t(\mathbf{x}|\mathbf{x}_0)$ because we do not want to distract readers from our actual generation method by spending too much time introducing and deriving this function that is totally not used. However, as pointed in this question by reviewer, we will add the above derivation into the appendix to ensure the integrity our framework.

• 3.2 Our alternative solution is Equation 25 and Algorithm 2, which works like the DDIM[2].

In short, our solution can be understood as deterministic reverse diffusion process or ODE with discrete time steps. We use a state transfer probability (Equation 25) $\tilde{p}([\tilde{\mathbf{x}}_ {t_ 1};\tau_ 1]|[\tilde{\mathbf{x}}_ {t_ 2};\tau_2],\mathbf{x}_ 0)$ to extend the traditional reverse process $p(\mathbf{x}_ {t_ 1}|\mathbf{x}_ {t_ 2},\mathbf{x}_ 0)$, where $1\leq t_1 < t_2 \leq T$. As in diffusion processes[3], if we set the timestep interval as 1, the probability path is obtained with $p_ t(\mathbf{x}|\mathbf{x}_ 0) = p(\mathbf{x}_ T|\mathbf{x}_ 0) \prod\limits_{s=t+1}^{T} p(\mathbf{x}_ {s-1}|\mathbf{x}_ {s},\mathbf{x}_ 0)$.

In our framework, we get $\tilde{p}_ t([\tilde{\mathbf{x}};\tau]|\mathbf{x}_ 0)=p([\tilde{\mathbf{x}}_ T;T]|\mathbf{x}_ 0)\prod\tilde{p}([\tilde{\mathbf{x}}_ {t_ i};\tau_ i]|[\tilde{\mathbf{x}}_ {t_ {i+1}};\tau_ {i+1}],\mathbf{x}_ 0)$.

Since the state transfer probability is a Dirac delta function, there is no randomness in the reverse process, where the $[\tilde{\mathbf{x}}_t;\tau]$ pair can be iteratively generated deterministically (Algorithm 2).

4. How to derive the training objective $\min ||\mathbf{x}_ 0-\mathbf{x}_ \theta||$?

• 4.1 From the perspective of the deterministic reverse process

We utilize the deterministic reverse diffusion process to generate data (Equation 25). Similar to DDPM[3], the learning objective for the unrescaled deterministic reverse diffusion process is derived in Appendix C.3 Equation 38, which can still be simplified to $||\mathbf{x}_ 0-\mathbf{x}_ \theta||$. For our rescaled reverse process $\tilde{p}([\tilde{\mathbf{x}}_ {t_\Delta};\tau_ \Delta]|[\tilde{\mathbf{x}}_ {t};\tau],\mathbf{x}_ 0)$, where both $\tau$ and $\tilde{\mathbf{x}}_ t$ are inputs, the objective is an equation set:

\left $ \begin{aligned} \mathbf{u}_ {\tau_ \Delta}\mathbf{x}_ 0 + \mathbf{v}_ {\tau_ \Delta}\hat{\boldsymbol{\epsilon}}&= \mathbf{u}_ {\tau_ \Delta} \mathbf{x}_ \theta + \mathbf{v}_ {\tau_ \Delta}\hat{\boldsymbol{\epsilon}}\\\\ \mathcal{T}(t-\Delta t,G(\mathbf{x}_ 0,\hat{\boldsymbol{\epsilon}})) &= \mathcal{T}(t-\Delta t,G(\mathbf{x}_ \theta,\hat{\boldsymbol{\epsilon}})) \end{aligned}\right $ \Rightarrow \left $ \begin{aligned} \mathbf{x}_ 0 &=\mathbf{x}_ \theta \\\\ G(\mathbf{x}_ 0,\hat{\boldsymbol{\epsilon}}) &= G(\mathbf{x}_ \theta,\hat{\boldsymbol{\epsilon}}) \end{aligned}\right $ ,

where $\hat{\boldsymbol{\epsilon}}=\Psi^{-1}([\tilde{\mathbf{x}}_ t,\tau])$ is a constant. Besides, $\tau_ \Delta$ is a constant for the above equation.

The unique solution to this equation set is $\mathbf{x}_\theta=\mathbf{x}_0$.

• 4.2 From the perspective of re-arranging

Given $\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0) = \left[\mathbf{u}'(\mathbf{x}_ 0,\mathcal{T}(t,G(\mathbf{x}_ 0,\boldsymbol{\epsilon})))\mathbf{x}_ 0+ \mathbf{v}'(\mathbf{x}_ 0,\mathcal{T}(t,G(\mathbf{x}_ 0,\boldsymbol{\epsilon})))\boldsymbol{\epsilon}\right]\frac{T-r\times G(\mathbf{x}_ 0,\boldsymbol{\epsilon})}{T}$, we can define $\tilde{v}_ \theta(\tilde{\mathbf{x}}_ t)=\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta)$. To get the $\min ||\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0) - \tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta)||$, we can direct calculate the partial derivate. For any $i$ dimension of $\mathbf{x}_\theta$, there is:

$$\frac{\partial ||\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0) - \tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta) ||}{\partial \mathbf{x}^i_ \theta} = \frac{\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0) - \tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta)}{||\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0) - \tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta) || } \frac{\partial \tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta) }{\partial \mathbf{x}^i_ \theta} =0$$

We can drop $\frac{\partial \tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta) }{\partial \mathbf{x}^i_ \theta}$, because it can not provide valid value of $\mathbf{x}_ \theta$ without the term of $\mathbf{x}_ 0$.

Solving the equation $\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0) - \tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta) = 0$ is difficult, but it is easy to prove that $\mathbf{x}_ \theta = \mathbf{x}_ 0$ is a solution.

Since $\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0)$ is deterministic, it is an injection function. This means that the same input will not produce different outputs. Therefore, $\mathbf{x}_ \theta = \mathbf{x}_ 0$ can always get the minimum value of $||\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0) - \tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta) ||$ and we simplify the objective to $\min ||\mathbf{x}_ 0 - \mathbf{x}_ \theta||$ for convenience and training stability.

Q2: Method and the probability contour

1. We have demonstrated the rescaled probability contours of $\tilde{p}_t(\tilde{\mathbf{x}}_t|\mathbf{x}_0)$ in Figure 2, where sample points are exactly calculated by our proposed method (Equation 22).

As illustrated in Figure 2, the probability contours calculated by our method can be easily adapted to different discrete boundaries and noising trajectories, which are inline with our expectations in Figure 1 of the Introduction Section.

Thank you for this suggestion and we will add a quick revision at the end of the Method Section and link our formula to the Figure 2.

2. $\tilde{p}_t(\mathbf{x}|\mathbf{x}_0)$ is a series of functions, where the consistency to the boundary contour is inversely proportional to the subscript $t$.

As illustrated in Figure 2 and Equations 21 and 22, $\tilde{p}_0(\mathbf{x}|\mathbf{x}_0)$ is exactly the boundary contour while $\tilde{p}_T(\mathbf{x}|\mathbf{x}_0)$ is a Gaussian distribution. When a boundary contour is very long in embedding space, far away point will gradually have a lower probability density under $\tilde{p}_t(\mathbf{x}|\mathbf{x}_0)$ as $t$ increases. This means, during the inference stage, the density probability will gradually be consistent with the contour when approaching the boundary, which is inline with Lines 46-51 and Figure 1B.

Besides, we will not face the extreme case where the boundary contour is an infinitely long straight line in practical application. Because the embedding space is a finite space and the values of the embedding points will be normalized into a finite range. Therefore, when we set the diffusion space a bit larger than the embedding space, there are always series of probability functions $\tilde{p}_t(\mathbf{x}|\mathbf{x}_0)$ from Gaussian distributions to the boundaries.

[1] Flow Matching for Generative Modeling

[2] Denoising Diffusion Implicit Models

[3] Denoising Diffusion Probabilistic Models

[4] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

[5] Rectified Flow: A Marginal Preserving Approach to Optimal Transport

[6] Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport

[7] Non-Autoregressive Neural Machine Translation

[8] Understanding Knowledge Distillation in Non-Autoregressive Machine Translation

[9] Analog Bits: Generating Discrete Data Using Diffusion Models with Self-Conditioning

Q3: Confidence factor

1. The confidence factor is named after some thoughts of sampling, such as the confidence interval. Our data is discrete and the uncertainty comes from the learned diffusion model.

Consider a simplified situation of only two tokens 'A' and 'B', which is the binary classification. When the confidence factor $r=1$, points $\mathbf{x}$ on the boundary of 'A' strictly follow $p(\mathbf{x}\in A)=p(\mathbf{x} \in B) = 0.5$. Since the learned diffusion model is expected to guide a random noise to this boundary, any subtal perturbation to the output of our learned diffuison model may lead to a reverse of the $\mathbf{x}$'s attribution. If we decrease the factor $r$ so that points on the boundary of the token 'A' have higher probability density, e.g., $p(\mathbf{x}\in A)=0.7>p(\mathbf{x}\in B)$, we will be more confident that our generated points of the learned diffusion model belong to the token 'A'. (In this situation, $p(\mathbf{x}\in A)\geq 0.7$ is the discrete area of A, $p(\mathbf{x}\in B)\geq 0.7$ is the discrete area of B, and $0.3 < p(\mathbf{x}\in A)< 0.7$ is the unattributed area.)

2. The mode collapse when $r=1$ comes from two part: One is the uncertainty of the learned diffusion model as described above. The other involves a traditional problem of diffusion processes.

• 2.1 From the perspective of uncertainty

Facing tasks with strong conditions, we are confident with the model's prediction since this is a easy task. Therefore, $r=1$ works perfect (Lines 227-229 and Table 1) as the strong conditions greatly reduce uncertainty of the learned model during inference.

When the condition is weak or there is no condition, uncertainty increases rapidly. This means that the sampled noisy training data constitutes a more difficult task with $r=1$. When the task difficulty exceeds the model's capability, the collapse occurs.

• 2.2 From the perspective of diffusion processes

This is also a problem similar to the well studied one in Flow Matching with Dynamic Optimal Transport [4,5,6], i.e., the path from the noise to target is too long and learning this path is hard. For example, suppose there are two unconditional targets 'A' and 'B', the sampled noisy data $\tilde{\mathbf{x}}_A$ of 'A' when $r=1$ may have a shorter path to 'B' than 'A'. Likewise, $\tilde{\mathbf{x}}_B$ can be closer to A. The diffusion models are required to map $\tilde{\mathbf{x}}_A$ to 'A' and $\tilde{\mathbf{x}}_B$ to 'B', but it is easier to learn the mapping of $\tilde{\mathbf{x}}_A$ to 'B'. This is a general problem for all diffusion models, not just ours. We believe the stability and performance of our method can be further improved with the Dynamic Optimal Transport, but we want to have a fair comparision with our baselines to demonstrate our effectiveness. Therefore, we just use a smaller $r$ to stabilize the training process and use the sub-optimal results to compare with the baselines.

Besides, the failure case is just the unexpected loss value during training, e.g., NaN, where these models are unable to generate contents.

3. Fixing the initial path means the Trajectory Alteration (Line 8 in Algorithm2) is optional in practice. This is a negligible detail of our model.

• 3.1 From the perspective of uncertainty

Line 8 in Algorithm 2 is to update the trajectory. If the learned diffusion model predicts a different $\mathbf{x}_0$, we have to update the original noise because currently we are in a different trajectory. If the learned model keeps changing the predicted target, which means it is uncertain about where to go, updating the trajectory correspond to an unreliable target may not be beneficial for the denoising process. If the learned model rarely changes its prediction, it means we are on the right path and no updates are needed. Therefore, it would be a viable option to discard this step (Line 8 in Algorithm 2) to make decoding faster.

• 3.2 From the perspective of experiments

As illustrated in Table 3, if we train with the rescaled forward process and decode with unrescaled trajectory, there will be only a slight performance degradation. The performance improvement of our model mainly comes from the training process, and some small changes in the decoding process may not have much impact. In practical application, we just discard the trajectory update (Line 8 in Algorithm 2) for simplicity in our experiments and find almost no performance degradation.

We sincerely appreciate your response, and we are delighted to have addressed some of your concerns. We hope to better align our contributions with your understanding.

Q1: loss function

$||\mathbf{x}_ 0 - \mathbf{x}_ \theta||$ is a simplified objective where we drop the coefficent (Equation 38 in Appendix C.3).

Let's rewrite the expectation $\mathbb{E}_{\tilde{\mathbf{x}}_t}[\tilde{u}_t(\tilde{\mathbf{x}}_t|\mathbf{x}_0)]$ in the sum form, we have:

$$\underbrace{\sum p(\tilde{\mathbf{x}}_ t) \tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0)}_ {\mathbb{E}_ {\tilde{\mathbf{x}}_ t}[\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0)]} = \sum p(\tilde{\mathbf{x}}_ t) \left[\mathbf{u}'\mathbf{x}_ 0+ \mathbf{v}'\boldsymbol{\epsilon}\right]\underbrace{\frac{T-r\times G(\mathbf{x}_ 0,\boldsymbol{\epsilon})}{T}}_ {0<\text{coefficient}<1} \leq \sum p(\tilde{\mathbf{x}}_ t) \left[\mathbf{u}'\mathbf{x}_ 0+ \mathbf{v}'\boldsymbol{\epsilon}\right] = \underbrace{\left[\mathbf{u}' \sum p(\tilde{\mathbf{x}}_ t) \mathbf{x}_ 0 + \mathbf{v}'\boldsymbol{\epsilon}\right]}_ {\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbb{E}_ {\tilde{\mathbf{x}}_ t}[\mathbf{x}_ 0])},$$

where $\mathbf{x}_ 0$ is the function of $\tilde{\mathbf{x}}_ t$ and $\frac{T-r\times G(\mathbf{x}_ 0,\boldsymbol{\epsilon})}{T}$ is a dynamic coefficient ranging from $0$ to $1$. Our current objective minimizes the upperbound of $\min \mathbb{E}||\tilde{u}_t - \tilde{u}^\theta_t||$, which is also effective.

As illustrated in Equation 38 of Appendix C.3, we simplify the training objective and drop the coefficient during training, which is similar to the equation 14 in DDPM. This coefficient works as to dynamically assign weights to different samples based on the noise and x0. Therefore, if we want to make $\mathbb{E}_ {\tilde{\mathbf{x}}_ t}[\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0)]=\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbb{E}_ {\tilde{\mathbf{x}}_ t}[\mathbf{x}_ 0])$, we need to calculate the dynamic coefficient during training, which may be time-consuming.

Q2: re-ranking

The transformer generates with a beam size of 5. We do not use sampling because

1. We strictly follow our baselines that all of them are using the beam search generation.
2. Diffusion LMs are not stable enough to keep achieving good results with simple sampling like the transformer, as illustrated in Appendix I and our baselines. They have the potential to generate high quality contents, but random initialization will have a great impact on the generated results. Transformers do not face this problem of randomness.

Take IWSLT14 as an example.

When we convert the beam search to sampling, there is a large drop for the reranked diffusion model. As illustrated in Table 1, we do not intend to claim that our method surpasses transformers, but rather to show that our method has the potential to produce results comparable to transformers in addition to outperforming existing diffusion models.

Q3: Table 3

We want to clarify that the most closest and simplest baseline is the DDIM, not the Flow Matching. Our method is a deterministic diffusion process or ODE with dicrete time step. We use the ODE framework to model the deterministic forward process (because DDPM or DDIM framework can not do this), so that our method is theoretically compatible with both the Diffusion process and Flow Matching. However, currently our method requires to predict $x_0$ in application and thus reverses the diffusion process step by step, which is actually a deterministic diffusion model, not a standard Flow Matching model. Therefore, the most closest and simplest baseline is the DDIM, not the Flow Matching. (The DDIM sampling of BitDiffusion is 11.37, slightly worse than Gaussian sampling)

If we want to apply the flow matching to the same discrete embedding space, there are several problems:

1. If we predict $x_0$, the only difference to BitDiffusion is the optimal transport trajectory. As in the ablation study (Table 2) for language, this sometimes improves performance, but the effect is not large.

Due to the high training cost, we currently only have results of 200K steps, where changing the trajectory to optimal transport does not make a significant difference.

2. If we predict $\tilde{u}$, our method is currently not adaptable to this objective. As demonstrated in Q1-3.1, we have to solve the equation $\tau=\mathcal{T}(t,G(\mathbf{x}_0,\frac{\mathbf{x}-\mathbf{u}(\mathbf{x}_0,\tau)\mathbf{x}_0}{\mathbf{v}(\mathbf{x}_0,\tau)}))$ to get the $\tau$ with respect to the change of $\mathbf{x}$ in real time, which is currently inefficient to apply.

We want to supplement that since solving the equation $\tau=\mathcal{T}(t,G(\mathbf{x}_0,\frac{\mathbf{x}-\mathbf{u}(\mathbf{x}_0,\tau)\mathbf{x}_0}{\mathbf{v}(\mathbf{x}_0,\tau)}))$ to get the $\tau$ with respect to the change of $\mathbf{x}$ in real time is difficult, this is why we use the equation 25 as an alternative. The equation 25 discretizes $t$ to finite steps and keeps track of previous $\tau$ to make it tractable. And therefore the discrete time step make our model the deterministic diffusion process rather than the ODE based flow matching with infinite continuous timesteps.

In addition, if the reviewer still wants to know the performance of Flow Matching on binary coding, (although our method is not closely related to Flow Matching and is currently not efficient to the Neural ODE framework in application), we are training the model with the TorchCFM framework and will provide the results before the end of the discussion period.

We sincerely appreciate your response, and we are delighted to have addressed most of your concerns. We attach great importance to your suggestions and conduct some analytical experiments for better understanding.

Q: Why our objective works?

1. In practice, errors come not only from the theory but also from the capability of neural networks. And the error caused by optimizing the upper bound is much smaller than the error from the neural network. Therefore, the theoretical error caused by optimizing the upper bound is negligible.

The symbols in the formula may sometimes be confusing, and substituting them into numbers will lead to a more intuitive understanding. We add an additional analysis experiment on IWSLT14 to reveal the thinking behind our formula.

We demonstrate the Expectation of $\mathbf{x}_ 0$'s and $\tilde{u}_ t$'s errors on the test set. It is easy to observe that, with the dynamic coeffficient $\frac{T-r\times G(\mathbf{x}_0, \boldsymbol{\epsilon})}{T}$, the value of $\mathbf{x}_0$'s error (8.4358) is much larger (than 1.5613). This supports Lines 180-182 that predicting $\tilde{v}_t^\theta$ will amplify the error on $\mathbf{x}_0$, whereas predicting $\mathbf{x}_0$ will reduce the error of neural networks on $\tilde{u}_t$. Therefore, predicting $\mathbf{x}_0$ is beneficial to reducing the impact of the prediction error of the neural network compared with $\tilde{u}_t$.

In addition, we convert the objective to $\Vert\tilde{u}_ t(\tilde{\mathbf{x}}_ t\vert\mathbf{x}_ 0) - \tilde{u}_ t(\tilde{\mathbf{x}}_ t\vert\mathbf{x}_ \theta)\Vert$ where the neural network still predicts $\mathbf{x}_0$. We find that the error expectations of $\mathbf{x}_0$ and $\tilde{u}_t$ decrease 0.0297 (0.35%) and 0.0095 (0.61%), respectively. This means that the error caused by optimizing the upper bound is basically negligible compared to the error of the neural network. Furthermore, predicting the upper bound has almost no impact on the final performance (0.07 of the BLEU score).

2. Discrete modeling may be more robust to neural network errors than the continuous modeling.

We compare the DDPM and BitDiffuison with different image embedding types over the eval set of cifar10.

For continuous embedding, each pixel is represented with a float number in [-1,1], where the predictions will be discretized to integers in [0,255]. It is really hard to recover the original pixels with only one step (1.18%). For binary coding, where the pixel representation is an 8-dimensional vector, it is more easy to recover $\mathbf{x}_0$ due to a larger absorption state (25.74%). This means the discrete embedding space can be more robust to errors of neural networks. (It's worth noting that although discrete embedding is more robust, it weakens the continuity between adjacent pixels and currently cannot exceed the performance of continuous models.)

We would like to express our sincere gratitude to you again. Your suggestions are of great help in improving the quality of our paper.

We believe the following minor revisions will improve the clarity of our paper.

1. In Training Objective part of Section 3.3, we will change the equation 24 to a quick introduction of the $\tilde{u}_ t$ and move the equation to this part. Then we will add the derivation from $\Vert\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ 0)-\tilde{u}_ t(\tilde{\mathbf{x}}_ t|\mathbf{x}_ \theta)\Vert$ to $\Vert\mathbf{x}_ 0-\mathbf{x}_ \theta\Vert$ as discussed above, including the upper bound. Therefore, our method covers the entire framework of the Flow Matching in theory.

2. In Reverse Process part of Section 3.3, we will add an explaination that solving the equation $\tau=\mathcal{T}(t,G(\mathbf{x}_0,\frac{\mathbf{x}-\mathbf{u}(\mathbf{x}_0,\tau)\mathbf{x}_0}{\mathbf{v}(\mathbf{x}_0,\tau)}))$ to get the $\tau$ with respect to the change of $\mathbf{x}$ in real time is difficult. Therefore, the theory and practice are distinguished in this part, where we explain how to implement our method.

3. In Section 4, we will add analyses about the expectation of errors as demonstrated above.

4. Other derivations and discussions that do not affect reading will be added to the appendix.

Q3: Boundaries of discrete areas

1. The discrete area of two distinct data points will not intersect.

As illustrated in Lines 34-37 and Equation 6, points in the discrete area will not be attributed to any other area by definition.

Take language modeling as an example, the discrete area of a token 'A' is the set of all points in the embedding space with the largest dot similarity to token 'A' than other tokens in the vocabulary. This means almost any point in the embedding space will be attributed to only one token. When the dot similarities of a point to different tokens equal, it means the point lies at the boundary of these discrete areas.

Besides, the discrete area is convex. This means all points within the boundary will only belong to this area, i.e., we can safely discribe the area with boundaries as in Section 3.1. The quick proof of convex is:

Suppose the embedding of token 'A' is $\mathbf{x}_0$, given two points $\mathbf{y}^0$ and $\mathbf{y}^1$ in the embedding space.

If $\mathbf{y}^0$ and $\mathbf{y}^1$ are inside the discrete area of token 'A' and for all embeddings $\mathbf{x}_i$ of other tokens,

there is: $\mathbf{x}_0\cdot\mathbf{y}^0\geq\mathbf{x}_i\cdot \mathbf{y}^0$ and $\mathbf{x}_0\cdot\mathbf{y}^1\geq\mathbf{x}_i\cdot \mathbf{y}^1$.

For any other points $\hat{\mathbf{y}} = t \ \mathbf{y}^0 + (1-t)\ \mathbf{y}^1, t\in[0,1]$,

there is $\mathbf{x}_0\cdot\hat{\mathbf{y}} = t(\mathbf{x}_0\cdot\mathbf{y}^0)+(1-t)(\mathbf{x}_0\cdot\mathbf{y}^1)\geq \mathbf{x}_i\cdot\hat{\mathbf{y}}$.

Therefore, the discrete area is convex.

2. Boundaries of two areas can overlap with the confidence factor $r=1$, but this is impossible when $r<1$.

When the confidence factor $r=1$, as illustrated in Equations 7 and 8, the boundary is where the probabilities of points belonging to the neighbour areas equal. Consider a simplified situation of only two tokens 'A' and 'B', which is the binary classification, the boundary of token 'A' overlaps the boundary of token 'B', where $p(\mathbf{x} \in A)=p(\mathbf{x}\in B) = 0.5$.

When the confidence factor $r$ decreases, the range of each discrete area is gradually reduced. For example, we may take $p(\mathbf{x} \in A)=0.7 > p(\mathbf{x}\in B)$ as the boundary of 'A' and $p(\mathbf{x} \in B)=0.7$ as the boundary of 'B'. In this case, $0.3<p(\mathbf{x} \in A)<0.7$ is an unattributed area and there is no overlap between the boundaries of 'A' and 'B'.

3. Pushing the noise to the boundary is beneficial.

• Empiricaly, previous work like Difformer and Dinoiser and our Table 1 have revealed that increasing the minimal noise scale can benefit the performance (Lines 217-219). In both their experiments and our Table 1, comparing to the original Diffusion LMs like DiffuSeq and SeqDiffuSeq, increasing the minimal noise scale (difformer and dinoiser) can improve the performance. And it is obvious that pushing the noise to the boundary is precisely what increases the minimal noise scale.

• Theoretically, probability density inside the discrete area can be ignored.

Let’s make a simple analogy. If we want to launch a satellite around the moon, we can treat the moon as a point mass. This is similar to the traditional continuous diffuison process. When it comes to landing on the moon, we have to take into account the shape of the moon, while the gravity field inside the moon does not matter. This is what our discrete problem looks like.

Based on the observation of the discrete area that any point inside this area will be attributed to the corresponding token (Lines 34-37), we can easily get the derivation: we don't need to model the probability density inside the discrete area because our diffusion model only needs to guide a noise to get into this area, where the accurate end position doesn't matter. Therefore, pushing the noise to the boundary does not cause theoretical damage to the probability modeling. And the dffusion model can be less disrupted by invalid data, i.e., points inside the area, during the training stage.

4. Certain perturbations can lead to inaccuracies in the final result without our confidence factor.

We use the confidence factor $r$ (Lines 137-147) to control the influence of perturbations. Given the above example of only two tokens 'A' and 'B'. When the confidence factor $r=1$, points $\mathbf{x}$ on the boundary of 'A' follow $p(\mathbf{x}\in A)=p(\mathbf{x} \in B) = 0.5$. Therefore, any perturbation to the prediction of the learned diffusion model may lead to a reverse of the attribution of $\mathbf{x}$. To alleviate this problem, we can decrease the factor $r$ so that points on the boundary have higher confidence, e.g., $p(\mathbf{x}\in A)=0.7>p(\mathbf{x}\in B)$, which is more robust to perturbations. In practice, $r=1$ works well for tasks with strong conditions and we should decrease $r$ for unconditional tasks.

We sincerely appreciate your meticulous review and the valuable comments. Please kindly find our response below.

Q1: Why can our framework enhance performance

1. From the perspective of Motivation

As illustrated in Introduction and Figure 1, we find that the continuous diffusion processes oversimplify the objective of discrete modeling. The discrete data are treated as single points, while they are actually areas in the diffusion space. This means diffusion models learned on the oversimplified objective generate the sub-optimal probability density contour, which is inconsistent with the discrete priors (Lines 38-41 and Figure 1A). Therefore, our method takes the discrete prior into consideration and design a more appropriate diffusion trajectory for discrete modeling problems (Lines 42-65 and Figure 1B).

2. From the perspective of Methodology

We theoretically derive the diffusion process conditioned on the discrete priors and obtain the sampling distribution of the corresponding forward process $\tilde{p}_t(\mathbf{x}|\mathbf{x}_0)$ (Equations 21 and 22). It is easy to calculate that the $\tilde{p}_0(\mathbf{x}|\mathbf{x}_0)$ is exactly the discrete boundary and $\tilde{p}_T(\mathbf{x}|\mathbf{x}_0)$ is a Gaussian distribution. During the inference stage, i.e., reversing the time step from $T$ to $0$, the density probability will gradually be consistent with the contour when approaching the boundary, which is inline with Lines 46-51 and Figure 1B. Hence, our method will precisely guide the random noise into the discrete area. Figure 2 demonstrates sample points exactly calculated by our proposed method (Equation 22) and the probability contours calculated by our method can be easily adapted to different discrete boundaries and noising trajectories, which are inline with our expectations in Figure 1 of the Introduction Section.

3. From the perspective of Experiment

Our framework is constructed on Difformer and BitDiffusion with the same configurations for both language and image generation tasks. As in Table 1 and 3, our method demonstrates significant improvements over them. Besides, ablations in table 4 reveal that, with the increase of the confidence factor $r$, i.e., increasing the influence of the discrete prior, the discrete modeling performance improves in all aspects.

Q2: Algorithm 1

You can consider $t\sim U[t_0,T]$. However, $t_0$ is not a constant but a random variable $t_0=G(\mathbf{x}_0,\boldsymbol{\epsilon})$ as illustrated in equation 12.

Sampling the $t$ from $U[1,T]$ and then transform it to $\tau$ with $\mathcal{T}(t,G(\mathbf{x}_0,\boldsymbol{\epsilon}))$ is equivalent to directly sampling $\tau$ from $U[G(\mathbf{x}_0,\boldsymbol{\epsilon}), T]$. We choose the former procedure because it is easy to perform in parallel. Algorithm 1 is just a commonly used element substitution technique in sampling.

Q3: Boundaries of discrete areas

Due to the character limit of the rebuttal, this is only an outline of our answer. Please refer to the comment for more details.

1. The discrete area of two distinct data points will not intersect.

As illustrated in Lines 34-37 and Equation 6, points in the discrete area will not be attributed to any other area by definition.

2. Boundaries of two areas can overlap with the confidence factor $r=1$, but this is impossible when $r<1$.

When the confidence factor $r=1$, as illustrated in Equations 7 and 8, the boundary is where the probabilities of points belonging to the neighbour areas equal. When the confidence factor $r$ decreases, the range of each discrete area is gradually reduced and there is no overlap between boundaries.

3. Pushing the noise to the boundary is beneficial.

• Empiricaly, previous work like Difformer and Dinoiser and our Table 1 have revealed that increasing the minimal noise scale can benefit the performance (Lines 217-219). And it is obvious that pushing the noise to the boundary is precisely what increases the minimal noise scale.

• Theoretically, probability density inside the discrete area can be ignored. Based on the observation of the discrete area that any point inside this area will be attributed to the corresponding discrete data (Lines 34-37), we can easily get the derivation: we don't need to model the probability density inside the discrete area because our diffusion model only needs to guide a noise to get into this area, where the accurate end position doesn't matter. Therefore, pushing the noise to the boundary does not cause theoretical damage to the probability modeling. And the dffusion model can be less disrupted by invalid data, i.e., points inside the area, during the training stage.

4. Certain perturbations can lead to inaccuracies in the final result without our confidence factor.

We use the confidence factor $r$ (Lines 137-147) to control the influence of perturbations. When we set a smaller $r$, the learned reverse process will be robust to perturbations.

Q4: Input of function $\Psi^{-1}([\cdot,\cdot])$

The input of $\Psi^{-1}([\cdot,\cdot])$ is valid for any arbitrary pair of point $\tilde{\mathbf{x}}_t$ and time $\tau$. Functions $\Psi$ and $\Psi^{-1}$ in Equation 14 are directly extended from Equations 4 and 5, which construct the invertible relationship between any point-time pair ($\tilde{\mathbf{x}}_t$-$\tau$) and their initial noise $\boldsymbol{\epsilon}$.

In Equation 14 of Section 3.1, we take the boundary point $\mathbf{x}_{t_0}$ and the crossing boundary time $t_0$ pair as input because we just want to estimate this boundary. However, this does not mean the input of $\Psi^{-1}$ can only be the pair of boundary point and crossing boundary time, while any other point-time pairs are still valid. This is why we use $\Psi^{-1}$ to update the trajectory in Line 8 of Algorithm 2.

We sincerely appreciate your response, and we are delighted to have addressed some of your concerns.

Q: The design of sampling (Algorithm 2)

Lets start from the sampling process of DDIM under our symbol denotion.

1. $\mathbf{x}_ T \sim \mathcal{N}(0,1)$
2. for $t=T,\dots,1$ do
3. $\hat{\mathbf{x}}_ 0 = \mathbf{x}_\theta(\mathbf{x}_t, t)$ (Pseudo Target)
4. $\boldsymbol{\epsilon} = \frac{\mathbf{x}_t - \mathbf{u}_t\hat{\mathbf{x}}_0}{\mathbf{v}_t}$ (Trajectory Alteration)
5. $\mathbf{x}_ {t-1} = \mathbf{u}_ {t-1}\hat{\mathbf{x}}_ 0 + \mathbf{v}_ {t-1}\boldsymbol{\epsilon}$ (Previous Sample)

This is a synchronous trajectory update where the algorithm directly updates the start point $\boldsymbol{\epsilon}$ of the trajctory based on the current prediction $\hat{\mathbf{x}}_0$.

Our method requires one more step of timestep rescaling $\tau = \mathcal{T}(t, G(\hat{\mathbf{x}}_ 0,\boldsymbol{\epsilon}))$, which takes the current trajctory start point as the input. However, the trajectory alteration (step 4 above) also requires the current timestep $\tau$ as the input $\hat{\boldsymbol{\epsilon}} = \frac{\tilde{\mathbf{x}}_ t - \mathbf{u}_ \tau\hat{\mathbf{x}}_ 0}{\mathbf{v}_ \tau}$. Therefore, our procedure can not be synchronous and we must choose $\tau$ or $\hat{\boldsymbol{\epsilon}}$ to be asynchronous.

As discussed with reviewer 1VFK (Q3.3 fixing the initial path), we find that even discarding the step of Trajectory Alteration leads to almost no performance degradation.

(1. From the perspective of uncertainty, if the learned diffusion model predicts a different $\mathbf{x}_0$, we have to update the original noise because currently we are in a different trajectory. If the learned model keeps changing the predicted target, which means it is uncertain about where to go, updating the trajectory correspond to an unreliable target may not be beneficial for the denoising process. If the learned model rarely changes its prediction, it means we are on the right path and no updates are needed. Therefore, it would be a viable option to discard this step of Trajectory Alteration to make decoding faster. 2. From the perspective of experiments, as illustrated in Table 3, if we train with the rescaled forward process and decode with unrescaled trajectory, there will be only a slight performance degradation. The performance improvement of our model mainly comes from the training process, and some small changes in the decoding process may not have much impact. In practical application, we just discard the trajectory update for simplicity in our experiments and find almost no performance degradation.)

Therefore, we choose $\hat{\boldsymbol{\epsilon}}$ to be asynchronous and the steps in the for loop (line 2) becomes:

3. $\hat{\mathbf{x}}_ 0 = \mathbf{x}_\theta(\tilde{\mathbf{x}}_t, t)$ (Pseudo Target) 4. $\tau = \mathcal{T}(t-1, G(\hat{\mathbf{x}}_ 0,\hat{\boldsymbol{\epsilon}}))$ (Trajctory Rescaling) 5. $\tilde{\mathbf{x}}_ {t-1}=\mathbf{u}_ \tau \hat{\mathbf{x}}_ 0 + \mathbf{v}_ \tau \hat{\boldsymbol{\epsilon}}$ (Previous Sample) 6. $\hat{\boldsymbol{\epsilon}} = \frac{\tilde{\mathbf{x}}_ {t-1} - \mathbf{u}_ \tau\hat{\mathbf{x}}_ 0}{\mathbf{v}_ \tau}$ (Asynchronous Trajectory Alteration)

where we add the Trajctory Rescaling step and move the Asynchronous Trajectory Alteration to the end. Other steps in Algorithm 2 is to keep track of current $\tau$ and $\hat{\boldsymbol{\epsilon}}$.

It's worth noting that we can not loop the $\tau$ as $t$ because $\tau$ is dynamic and different for each pair of $\mathbf{x}_0$ and $\boldsymbol{\epsilon}$. In the same iteration round, different $\mathbf{x}_0$ corresponds to different $\tau$.

Therefore, Algorithm 2 can complete the iterative prediction of $\mathbf{x}_0$ with subtal errors. And in the analytical experiments discussed with the reviewer 1VFK, the errors in the sampling stage have almost no impact on the final results.

(There is a typo in Algorithm 2 that $\tilde{\mathbf{x}}_ t$ in Line 6 should be $\tilde{\mathbf{x}}_ {t-\Delta t}$)

We hope to supplement the above description in a more vivid way.

The sampling process of DDIM can be demonstrated as:

$$\begin{aligned} \mathbf{x}_ \theta ^{t+1} & \\\\ \searrow &\\\\ \boldsymbol{\epsilon} \rightarrow &\mathbf{x}_ t \end{aligned} \Rightarrow \begin{aligned} &\quad \mathbf{x}_ \theta ^ {t}\\\\ &\nearrow \\\\ \boldsymbol{\epsilon} \rightarrow &\mathbf{x}_ t \end{aligned} \Rightarrow \begin{aligned} &\qquad \ \mathbf{x}_ \theta ^ t\\\\ & \qquad \ \downarrow \\\\ \boldsymbol{\epsilon} \rightarrow &\mathbf{x}_ t \rightarrow \boldsymbol{\epsilon}' \end{aligned} \Rightarrow \begin{aligned} &\qquad \ \mathbf{x}_ \theta ^ t\\\\ & \qquad \ \downarrow \quad \searrow \\\\ \boldsymbol{\epsilon} \rightarrow &\mathbf{x}_ t \rightarrow \boldsymbol{\epsilon}' \rightarrow \mathbf{x}_ {t-1} \end{aligned}$$

In this process, the reverse trajectory is $\boldsymbol{\epsilon} \rightarrow \mathbf{x}_ t \rightarrow \mathbf{x}_ 0$. When the prediction of $\mathbf{x}_ 0$ changes, the trajectory is updated with $\mathbf{x}_ t$ as the fixed point $\boldsymbol{\epsilon}' \rightarrow \mathbf{x}_ t \rightarrow \mathbf{x}_ \theta$, which can be demonstrated as:

$$\begin{aligned} \boldsymbol{\epsilon} \rightarrow &\mathbf{x}_ t \rightarrow \mathbf{x}_ 0 \end{aligned} \Rightarrow \begin{aligned} &\qquad\ \mathbf{x}_ \theta\\\\ &\quad \nearrow \\\\ \boldsymbol{\epsilon} \rightarrow &\mathbf{x}_ t \rightarrow \mathbf{x}_ 0 \\\\ \nearrow&\\\\ \boldsymbol{\epsilon}' \quad\ & \end{aligned} \Rightarrow \begin{aligned} &\qquad\qquad\ \mathbf{x}_ {\theta}\\\\ &\qquad \quad \nearrow \\\\ &\qquad\ \mathbf{x}_ {t-1}\\\\ &\quad \nearrow \\\\ \boldsymbol{\epsilon} \rightarrow &\mathbf{x}_ t \rightarrow \mathbf{x}_ 0 \\\\ \nearrow&\\\\ \boldsymbol{\epsilon}' \quad\ & \end{aligned}$$

Therefore, DDIM algorithm can be simplified as:

1. $\mathbf{x}_ T \sim \mathcal{N}(0,1)$
2. for $t=T,\dots,1$ do
3. $\hat{\mathbf{x}}_ 0 = \mathbf{x}_\theta(\mathbf{x}_t, t)$ (Pseudo Target)
4. $\mathbf{x}_ {t-1} = \mathbf{u}_ {t-1}\hat{\mathbf{x}}_ 0 + \mathbf{v}_ {t-1}\frac{\mathbf{x}_t - \mathbf{u}_t\hat{\mathbf{x}}_0}{\mathbf{v}_t}$ (Previous Sample)

In our algorithm 2, the timestep rescaling $\tau = \mathcal{T}(t, G(\mathbf{x}_ 0, \boldsymbol{\epsilon}))$ takes both $\mathbf{x}_ 0$ and $\boldsymbol{\epsilon}$ as the input. This means errors in the predicted $\mathbf{x}_ \theta$ will be maginified if we update $\boldsymbol{\epsilon}$ before rescaling the timestep. Therefore, there will be two different solutions:

1. One is our implemention, where we directly discard the Trajectory Alteration step, and the fixed point of our reverse trajectory is $\boldsymbol{\epsilon}$:

$$\begin{aligned} \boldsymbol{\epsilon} \rightarrow &\tilde{\mathbf{x}}_ t \rightarrow \mathbf{x}_ 0 \end{aligned} \Rightarrow \begin{aligned} &\qquad \qquad \mathbf{x}_ \theta\\\\ &\qquad \ \ \ \nearrow\\\\ &\qquad \tilde{\mathbf{x}}_ t' \\\\ &\ \ \nearrow \\\\ &\boldsymbol{\epsilon} \rightarrow \tilde{\mathbf{x}}_ t \rightarrow \mathbf{x}_ 0 \\\\ \end{aligned}\Rightarrow \begin{aligned} &\qquad\qquad\qquad \mathbf{x}_ \theta\\\\ &\qquad\qquad \ \ \ \nearrow\\\\ &\qquad \qquad \tilde{\mathbf{x}}_ {t-1}\\\\ &\qquad \ \ \ \nearrow\\\\ &\qquad \tilde{\mathbf{x}}_ t' \\\\ &\ \ \nearrow \\\\ &\boldsymbol{\epsilon} \rightarrow \tilde{\mathbf{x}}_ t \rightarrow \mathbf{x}_ 0 \\\\ \end{aligned}$$

It is worth noting that, in this situation, the Trajectory Alteration step is just like a place holder and does not change the value of $\boldsymbol{\epsilon}$, because the Previous Sample (Step 5) and Asynchronous Trajectory Alteration (Step 6) are exaclty the same equation.

2. The Asynchronous Trajectory Alteration works on the other solution where we still take $\tilde{\mathbf{x}}_ t$ as the fixed point. In this situation, the Previous Sample step is $\tilde{\mathbf{x}}_ {t-1} = \mathbf{u}_ \tau\hat{\mathbf{x}}_ 0 + \mathbf{v}_ \tau \frac{\tilde{\mathbf{x}}_ t-\mathbf{u}_ {\tau_ \text{prev}}\hat{\mathbf{x}}_ 0}{\mathbf{v}_ {\tau_ \text{prev}}}$. This means we calculate the $\tau$ with $\boldsymbol{\epsilon}$ but generate $\tilde{\mathbf{x}}_ {t-1}$ with the updated $\boldsymbol{\epsilon}'$. The corresponding reverse trajectory is:

$$\begin{aligned} \boldsymbol{\epsilon} \rightarrow &\tilde{\mathbf{x}}_ t \rightarrow \mathbf{x}_ 0 \end{aligned} \Rightarrow \begin{aligned} &\qquad\ \mathbf{x}_ \theta\\\\ &\quad \nearrow \\\\ \boldsymbol{\epsilon} \rightarrow &\tilde{\mathbf{x}}_ t \rightarrow \mathbf{x}_ 0 \\\\ \nearrow&\\\\ \boldsymbol{\epsilon}' \quad\ & \end{aligned} \Rightarrow \begin{aligned} &\qquad\qquad\ \mathbf{x}_ {\theta}\\\\ &\qquad \quad \nearrow \\\\ \boldsymbol{\epsilon} &\rightarrow \ \tilde{\mathbf{x}}_ {t-1}\\\\ &\quad \nearrow \\\\ \boldsymbol{\epsilon} \rightarrow &\tilde{\mathbf{x}}_ t \rightarrow \mathbf{x}_ 0 \\\\ \nearrow&\\\\ \boldsymbol{\epsilon}' \quad\ & \end{aligned}$$

Thank you for your patience and kindness, your suggestions and guidance have been very helpful.

We will make the following minor revisions to make it clearer.

1. Replace the current $\Delta t$ iteration into an easier notation.
2. Add the explaination about the Asynchronous Conflict between $\tau$ and $\epsilon$ and trajectory alteration strategy as discussed above.
3. Add comparison with DDIM in the appendix for better understanding.
4. Other derivations and discussions that do not affect reading will be added to the appendix.

We believe these revisions can improve the clarity of our algorithm and we would like to express our sincere gratitude to you again.

Besides, the asynchronous conflict problem is the same as the problem of $\tau=\mathcal{T}(t,G(\mathbf{x}_0,\frac{\mathbf{x}-\mathbf{u}(\mathbf{x}_0,\tau)\mathbf{x}_0}{\mathbf{v}(\mathbf{x}_0,\tau)}))$ discussed with reviewer 1VFK.

In Reverse Process part of Section 3.3, we will add an explaination that solving the equation $\tau=\mathcal{T}(t,G(\mathbf{x}_0,\frac{\mathbf{x}-\mathbf{u}(\mathbf{x}_0,\tau)\mathbf{x}_0}{\mathbf{v}(\mathbf{x}_0,\tau)}))$ to get the $\tau$ with respect to the change of $\mathbf{x}$ in real time is difficult. Therefore, the theory and practice are distinguished in this part, where we explain how to implement our method.

Therefore, we will combine these two explanations.