Task-Guided Biased Diffusion Models for Point Localization

1. The necessity of applying diffusion models to point localization tasks is not illustrated. In other words, the positive properties of diffusion models seem not incorporated into this approach. Whether the diffusion model here can be replaced with other simple network for the same purpose while the posed two problems are correspondingly absent.
2. The proposed method is not correlated to the task of point localization, as the targeted problems are general for diffusion models. So why the point localization tasks are chosen.
3. Lack of the visualization for two out of three comparison tasks (neither in appendix).
4. Introducing the task-biased noise in the forward diffusion process have been proposed in literatures [1][2]. The comparison baselines should include these methods instead of all non-diffusion methods and the discussion is recommended.
5. Please carefully check the full manuscript for writing typos. [1] Accelerating Diffusion Models for Inverse Problems through Shortcut Sampling [2] Image Restoration with Mean-Reverting Stochastic Differential Equations

• While the biased forward diffusion process is defined and derived, the inverse $q(x_{t-1}|x_t,x_0,\tilde{x})$ is not, which normally would allow the definition of the estimated generative process which is not included in the paper. Furthermore, the authors do not provide the KL divergence between $p(x_{t-1}|x_t,\tilde{x})$ and $q(x_{t-1}|x_t,x_0,,\tilde{x})$ which would allow the construction of the loss function for the biased conditional diffusion. Only the loss for a vanilla diffusion model is given in Equation (2) (also in $L_n$, n is not defined, it should be $L_{simple}$?). The score estimator $\epsilon_{\theta}$ in loss function in (2) does not admit any dependent variables to condition on. To summarise, no details are provided with regards to the generative process.
• The task-guided loss is not motivated and explained properly. In addition, the expression of the loss $L_t$ is not provided explicitly anywhere in the paper. Furthermore, the ground truth image cannot be reconstructed with the expression (9), and one should notice that the score here is not defined to depend on any conditioned variables either. When $L_t$ is the MSE between $\tilde{x_0}$ and $x_0$, then $L_t$ forces the model to simply try to predict a scaled version of the data point $x_0$. How do the authors justify training the same network to predict both the score and the data point? Training a network to learn one of these enables us to define a function predicting the other (they both enable us to predict the mean of $q(x_{t-1}|x_t,x_0)$, with some modifications), but training the same network to perform both tasks is counter-intuitive to me. In [1] (Equation 16) a similar combination of the loss is used, but here one network focuses on modelling the mean and the other the covariance matrix of $p(x_{t-1}|x_t)$.
• The authors appear to use the classical DDPM from Ho et al 2020, and the classical generative procedure as a baseline, that is, GL+DDPM (and probably in their method, even though that is not described). Considering that the suboptimal classical DDPM generative procedure is used for the baseline, it would be interesting to see the comparison between GL+EDM (which is more efficient [2]) and the model the authors introduce.

Minor Weaknesses:

• In Equation (1), it should be $\sqrt{1-\bar{\alpha}_t}$ in front of the noise instead of $1-\sqrt{\bar{\alpha}_t}$. Also, in Equation (9) $\sqrt{1-\bar{\alpha}_t}$ and $\sqrt{\bar{\alpha}_t}$ should switch places.
• Some parts of the text can be expressed better (example: third line of the abstract).
• The structure is not ideal. The details of diffusion models should be given in Section 2 under a subsection titled 'background' and not in Section 3. In addition, the position of tables/figures and the places when they are referenced should be improved. Figure 3 is referenced in the introduction, but is located in page 5. While Figure 1 is located in page 3, but is referenced for the first time at the end of page 4. Similarly, Table 1, is placed in page 6, but is referenced for the first time in page 8, while table 2 is introduced and referenced in page 7. This makes reading the paper quite confusing.
• A table with all the steps performed should be given as comparing them just from written text in the current form is difficult.

However, I still have some concerns about this paper:

1. The proposed method somehow lacks novelty here. In general, the author just utilizes existing methods and with a "Diffusion model" to build the final model predictions. To me, it is like a two-stage/post-processing step of the model. Also, the proposed method is somehow simple: The task loss is not a new component in the model, the author just applies it in the training process of the diffusion model. Also, the biased term is not a brand new idea in the vision community (like non-gaussian diffusion models and others).

2. The "biased" DDPM needs careful design: As I said before, the proposed biased diffusion model is just a diffusion model with prior. The computer vision community has multiple choices to inject priors to the diffusion models like conditions, texture inversions, etc. The author could provide more choices in the ablation step.

3. The experiments part needs to be revised. From my side, the author could have a brief recap of diffusion models in 3.1, then give more space to provide ablations, and more detailed methods (like the proposed framework on different methods in the same task), FPS, training times, etc.

• The motivation and hypothesis of the paper may be questionable: At the beginning of the abstract, the author “hypothesize that diffusion models can be used to enhance the performance of deep learning methods for predictive tasks involving sparse outputs, such as point-localization tasks.” Where is this hypothesis or intuition from? What kind of data distribution is learned by the diffusion model in this formulation? Why should the incorporation of a diffusion model be helpful for the sparse outputs? Although Sec. 3.2.1 describes some motivations, it is not very convincing and clear why the diffusion model is inherently helpful in this case.
• About the novelty of task-guided bias: In order to reduce sampling steps, the method uses the baseline prediction as initial prediction. This looks very similar to a bunch of previous works on conditional diffusion models, such as [1], while the paper seems not to mention any previous works in this line and no comparison as well. From Sec. 3.2.2, it looks like the forward and backward diffusion process all needs a biased noise distribution, does it mean that the training process of the diffusion model also requires the paired data (initial prediction and ground truth)? This may indicate that for each different dataset/task/baseline to obtain the initial prediction, it is required to train a new diffusion model? What kind of data distribution priors are learned in the process?
• About the purpose of task-guided loss: From the paper, this loss is proposed to reduce the variance of prediction from different noise so as to improve accuracy. First, it is not very clear to me why minimizing the stochasticity of the reconstructed samples can help with accuracy. Second, with such constraint, it seems the prior Gaussian noise distribution is collapsed or converted to a delta data distribution? What does this mean in the predictive task? If the stochasticity of the reconstructed samples is not preferred, why not just use a fixed input or deterministic sampling process?
• Experiments and results: 1) The proposed method seems to be quite sensitive to the baseline method used to get the initial prediction, as shown in Table 4. 2) The gain and difference between the proposed method compared with corresponding baseline models seem to be relatively small, as shown in Table 4. 3) In Table 1, the precision of the baseline method achieves a much higher score than all other DDPM incorporated methods. Again, what prior has been learned by such a diffusion model and why it should benefit the predictive results. The motivation is not very clear.
• One of the many contribution claims of the paper is that task-guided bias can improve sampling efficiency. What is the sampling time of the proposed method? How does it compare with the model without task-guided bias? How does it compare with the baseline models in those tables achieving comparable results?
• No discussion about the method limitation in the paper.
• The figure demonstration such as Figure 1 could be further improved.

[1] I2SB: Image-to-Image Schrödinger Bridge, ICML 2023.