RODS: Robust Optimization Inspired Diffusion Sampling for Detecting and Reducing Hallucination in Generative Models

W&Q1: Related works and Comparison

Thank you for pointing out these related works. We will include citations and a brief discussion of [1]–[3] in the revised version.

While [1] ("Constrained Diffusion with Trust Sampling") uses the norm of the predicted noise, aligned with the score direction, as a constraint for update validity, our approach instead focuses on the norm of local changes in the score field to guide updates. Given the differing objectives and mechanisms, a direct comparison is not applicable.

Regarding [2] (“Mitigating Hallucinations through Adaptive Attention Modulation”), it adopts a different strategy—modulating attention during sampling—while our approach focuses on trajectory-level geometric regularization. As the paper was posted on arXiv in February 2025 and no code was available at the time of our submission, a direct experimental comparison was not feasible.

As for [3] (“Understanding Hallucinations through Mode Interpolation”), we do include a comparison in Section 4.1 and Appendix C.3. Their method detects hallucinations via temporal instability along the denoising trajectory, whereas our approach targets spatial-domain instability and further enables correction. Additionally, their released code (for now) only covered 1D&2D Gaussian and SIMPLE SHAPES dataset, which could not be naturally transferred to our datasets. We plan to provide a fair experimental comparison once their full implementation becomes available.

We appreciate the reviewer’s emphasis on thorough comparison. As noted in Section 5, due to substantial differences across prior works in methodology, assumptions, and evaluation protocols, there is currently no standardized benchmark for hallucination detection and correction. We are committed to releasing our code and annotated data to facilitate future research and establish a common basis for meaningful comparison across methods.

---

Q2: Evaluation on background blurry problem

Thank you for pointing out the background blurrier phenomenon. This occurs because it is hard to distinguish between high-frequency textures and hallucinations. In our generation, the main subjects remain sharp and artifact-free, which is precisely the goal of our correction. The slight background smoothing does not compromise the diversity of the generation or the sharpness of the primary objects.

To validate such idea, we have further computed standard image-quality and diversity metrics on 1,000 unconditional samples from FFHQ, AFHQv2, and 11k-hands. The table below reports FID, Inception Variance [1] (which quantifies variability in the latent representations), and DreamSim Diversity [2] (which measures the variance of features extracted from a batch of generated images, and our batch size is 64).

These results show that RODS-CAS preserves diversity while keeping FID comparable. In our generation, the main subjects remain sharp and artifact-free. This is further supported by the 11k-hands experiment, where our method improves hand confidence (HAND-CONF) from 0.9216 (EDM-Euler) and 0.9227 (EDM-Heun) to 0.9347 (RODS-CAS). HAND-CONF is computed by running a pre-trained hand detector [3] and calculating the confidence scores of the detections. This metric is widely used to assess the anatomical plausibility of generated hands. Because hands are structurally well-defined and challenging to synthesize, this benchmark provides a sensitive and interpretable proxy for visual realism.

[1] Miao, Zichen, et al. "Training diffusion models towards diverse image generation with reinforcement learning." CVPR. 2024. [2] Domingo-Enrich, Carles, et al. "Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control." arXiv preprint 2024. [3] Zhang, Fan, et al. "Mediapipe hands: On-device real-time hand tracking." arXiv preprint 2020.

---

L: Limitation on current dataset

Thank you for raising this up. In the unconditional generation scenario, we have tested RODS on well-established benchmarks (AFHQv2, FFHQ, 11k-hands), where hallucinations can be easily measured without domain expertise. We acknowledge that in highly diverse or specialized domains, such as medical imaging, the definition of “hallucination” must be grounded in expert judgment. We are collaborating with domain practitioners to evaluate RODS under these conditions.

Dear Reviewer UPcW,

Thank you for your thoughtful review and for the time you’ve invested in our paper. We’ve gone through your feedback and responded to each point in the rebuttal above.

Please let us know whether our responses have resolved your concerns, especially with the discussion period ending in two days. If anything remains unclear or you’d like further clarification, we’d be happy to continue the discussion. And if our responses have resolved your concerns, we would greatly appreciate your reconsideration of our score.

W: Explain why / when RODS reduce hallucinations

Thank you for pointing this out. Below we summarize the key take‑aways and point to the detailed explanations we have now added (see the two responses to the questions below).

1. Sharp local curvature ⇒ high score uncertainty ⇒ higher hallucination risk (Q1)

2. Our look‑ahead worst‑case update provably lowers the maximum one‑step error in exactly those high‑risk zones while (i) score locally Lipschitz, (ii) score error bounded by $\rho$, (iii) step stays inside the ball. (Q2)

---

Q1: Connection between Curvature, Uncertainty, and Hallucination

Thank you for asking us to clarify the connection between curvature, uncertainty, and hallucination. The key idea is that hallucination correlates with model uncertainty [1], which can be quantitatively captured by the “rapid fluctuations (curvature)” of the density field.

Curvature is a proxy for model uncertainty

• Score field. We denote the learned score by $v(x) = \nabla_x \log P_t(x)$.
• Curvature index. As stated in §4.1 (Eq. 8), we define our curvature index is:

$$H(x) = \|\nabla_x \|v(x + \delta)\| - \nabla_x \|v(x)\|\|, \quad \delta = \arg \max_{\|\delta\| = \rho} \|v(x + \delta)\|,$$

Because $H(x)$ compares the gradient of $\|v\|$ at two nearby points, it measures second-order variation of the density field, i.e., how smoothly the score field changes in space. This reflects the key idea that hallucination correlates with model uncertainty [1]. By probing these rapid, spatial fluctuations, i.e., second‑order changes or “curvature”, we obtain a quantitative signal for potential sampling failures. In Appendix A.2, we further describe the conceptual foundation in detail:

“In well-behaved regions, this vector field changes smoothly. But in low-density areas, $v(x)$ can shift direction quickly—these are the regions where hallucinations are likely to occur.” (line 467–471)

“Rapid changes in this quantity often indicate instability in the local geometry of the score function.” (line 475–476)

---

Q2: Connection between optimization against the worst-case scenario and robustness

This follows a common principle in robust optimization: by guarding against the worst-case scenario, we ensure robust and reliable performance across all admissible uncertainties. If loss can be reduced even under the most adverse local perturbation, then all other nearby cases are guaranteed to be no worse. This approach is valid when the score function is locally smooth, the uncertainty is bounded, and the step size remains small. Below, we provide a detailed analysis to explain why it enhances robustness and under what conditions the guarantee holds.

1. What the algorithm actually does

Key idea : we peek at the place where the model is most uncertain, borrow a direction that corrects that uncertainty, and use it to update the current sample.

2. Why this look‑ahead step is robust - from robust optimization perspective

$$\min_x f_t(x + \delta),\ \text{ where } \ \delta = \arg\max_{\substack{\delta' \in \mathcal M}} \hat f_t(x, \delta').$$

• Inner maximization ( $\delta(x)$ )
  Chooses the perturbation inside the admissible set $\mathcal M$ that makes the surrogate loss $\hat f_t$ worst.
  ➜ Find the scenario where the current score estimate is least trustworthy.

• Outer minimization
  Once that worst case is fixed, we ask: “What move of $x$ will reduce that worst loss the most?”
  The answer is to step along the negative gradient at the worst point (our $d$). This follows standard convex duality: the steepest descent direction minimises the first‑order approximation of $f_t$ under that fixed worst perturbation.

• Why this is safe
  Because the score field is Lipschitz‑continuous, every other perturbation in $\mathcal M$ produces a loss that is no worse than the one at $\delta(x)$. Therefore, a direction that decreases the loss at the worst perturbation will decrease (or at least not increase) the loss for all admissible perturbations, i.e., the surrounding uncertainties.

3. Conditions for the guarantee

• Smooth nearby score – The score network is locally Lipschitz (smooth in a small neighborhood).
• Error is bounded – We assume the network’s score is never “too far off” from the true score inside the small ball we inspect.
• Step stays small – The time step $\Delta t$ is small enough for the second‑order Taylor expansion to remain valid.

Under these three assumptions the look‑ahead worst‑case update provably minimises the one‑step worst‑case loss and, as shown in Appendix A, yields more robust sampling in practice.

---

Q3: Clarification on Algorithm 2

Thanks for asking. The standard diffusion sampler solves $\min_x\ f_t(x).$ RODS instead solves the following bilevel problem (introduced right after line 232), which is the core of Algorithm 2:

$$\min_x f_t(x + \delta),\ \text{ where } \ \delta = \arg\max_{\substack{\delta' \in \mathcal M}} \hat f_t(x, \delta').$$

Two cases we addressed in the paper are:

1. Sharpness‑Aware Sampling (SAS) (lines 219–223): $\hat f_{t,\delta'}(x,\delta') = f_t\bigl(x + \delta'\bigr).$
2. Curvature‑Aware Sampling (CAS) (line 224): $\hat f_{t,\delta'}(x,\delta') = \bigl ||\nabla f_t(x + \delta')\bigr ||.$

Algorithm 2 simply plugs in the chosen $\hat f_{t,\delta'}$ once the curvature detector $H(x_t)\geq\epsilon$ signals a “high‑risk” step, and then takes the corresponding robust update.

---

Q4: Evaluation on image quality

Thank you for encouraging a broader quality assessment. We have therefore computed standard image-quality and diversity metrics on 1,000 unconditional samples from FFHQ, AFHQv2, and 11k-hands. The table below reports FID, Inception Variance [1] (which quantifies variability in the latent representations), and DreamSim Diversity [2] (which measures the variance of features extracted from a batch of generated images, and our batch size is 64).

These results show that RODS-CAS preserves diversity while keeping FID comparable. The modest FID rise is largely explained by background smoothing: our curvature test sometimes removes ambiguous, high-frequency textures in the complex backgrounds, which the FID metric interprets as loss of detail. However, such effect could be analogous to portrait-mode photography, where a deliberately blurred background can worse a texture-based metric (FID) without reducing perceived quality. In our generation, the main subjects remain sharp and artifact-free, which is crucial as it is exactly the goal of our correction. This is further supported by the 11k-hands experiment, where our method improves hand confidence (HAND-CONF) from 0.9216 (EDM-Euler) and 0.9227 (EDM-Heun) to 0.9347 (RODS-CAS). HAND-CONF is computed by running a pre-trained hand detector [3] and calculating the confidence scores of the detections. This metric is widely used to assess the anatomical plausibility of generated hands. Because hands are structurally well-defined and challenging to synthesize, this benchmark provides a sensitive and interpretable proxy for visual realism. Overall, RODS-CAS delivers equally diverse, comparably high-quality unconditional samples while reducing hallucinations.

[1] Miao, Zichen, et al. "Training diffusion models towards diverse image generation with reinforcement learning." CVPR. 2024. [2] Domingo-Enrich, Carles, et al. "Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control." arXiv preprint 2024. [3] Zhang, Fan, et al. "Mediapipe hands: On-device real-time hand tracking." arXiv preprint 2020.

W1&Q2: Evaluation on image quality

Thank you for encouraging a broader quality assessment. We have therefore computed standard image-quality and diversity metrics on 1,000 unconditional samples from FFHQ, AFHQv2, and 11k-hands. The table below reports FID, Inception Variance [1] (which quantifies variability in the latent representations), and DreamSim Diversity [2] (which measures the variance of features extracted from a batch of generated images, and our batch size is 64). Because RODS adjusts only the sampling trajectory and never alters the trained score network, its negative log-likelihood (NLL) is identical to that of the baseline sampler and is omitted for brevity.

These results show that RODS-CAS preserves diversity while keeping FID comparable. The modest FID rise is largely explained by background smoothing: our curvature test sometimes removes ambiguous, high-frequency textures in the complex backgrounds, which the FID metric interprets as loss of detail. However, such effect could be analogous to portrait-mode photography, where a deliberately blurred background can worse a texture-based metric (FID) without reducing perceived quality. In our generation, the main subjects remain sharp and artifact-free, which is crucial as it is exactly the goal of our correction. This is further supported by the 11k-hands experiment, where our method improves hand confidence (HAND-CONF) from 0.9216 (EDM-Euler) and 0.9227 (EDM-Heun) to 0.9347 (RODS-CAS). HAND-CONF is computed by running a pre-trained hand detector [3] and calculating the confidence scores of the detections. This metric is widely used to assess the anatomical plausibility of generated hands. Because hands are structurally well-defined and challenging to synthesize, this benchmark provides a sensitive and interpretable proxy for visual realism. Overall, RODS-CAS delivers equally diverse, comparably high-quality unconditional samples while reducing hallucinations.

Additional conditional metrics such as CLIP-score are not applicable here, as we work in the unconditional setting. We will incorporate this table and the accompanying discussion into the revised manuscript to present a more complete picture of image quality alongside hallucination reduction.

[1] Miao, Zichen, et al. "Training diffusion models towards diverse image generation with reinforcement learning." CVPR. 2024. [2] Domingo-Enrich, Carles, et al. "Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control." arXiv preprint 2024. [3] Zhang, Fan, et al. "Mediapipe hands: On-device real-time hand tracking." arXiv preprint 2020.

---

W2&Q1: Connection between curvature, uncertainty, and hallucination

Thank you for asking us to clarify the connection between curvature, uncertainty, and hallucination. The key hypothesis is that hallucination correlates with model uncertainty [1], which can be quantitatively captured by the “rapid fluctuations (curvature)” of the density field. Experimentally, we showed that the hallucination is not caused by various step size (Figures 9–10), but rather a directional error within low-density, high-uncertainty regions: merely changing the step-size of the sampler does not help if the direction itself is unreliable.

Curvature is a proxy for model uncertainty

• Score field. We denote the learned score by $v(x) = \nabla_x \log P_t(x)$.
• Curvature index. As stated in §4.1 (Eq. 8), we define our curvature index is:

$$H(x) = \|\nabla_x \|v(x + \delta)\| - \nabla_x \|v(x)\|\|, \quad \delta = \arg \max_{\|\delta\| = \rho} \|v(x + \delta)\|,$$

Because $H(x)$ compares the gradient of $\|v\|$ at two nearby points, it measures second-order variation of the density field, i.e., how smoothly the score field changes in space. This reflects the key idea that hallucination correlates with model uncertainty [1]. By probing these rapid, spatial fluctuations, i.e., second‑order changes or “curvature”, we obtain a quantitative signal for potential sampling failures. In Appendix A.2, we further describe the conceptual foundation in detail:

“In well-behaved regions, this vector field changes smoothly. But in low-density areas, $v(x)$ can shift direction quickly—these are the regions where hallucinations are likely to occur.” (line 467–471)

“Rapid changes in this quantity often indicate instability in the local geometry of the score function.” (line 475–476)

Why “just shrink the step” is insufficient

Intuitively, shrinking the step size is a natural approach. We evaluated a naive fix, using a much smaller step size by running an EDM-Euler sampler with 200 uniform steps (Figures 9–10). Although each step is five times shorter than the 40-step baseline, hallucinations still appear. In contrast, in tons of our experiments, we often observed that RODS-CAS actively changes the update direction whenever the curvature index is high and succeeds in removing most of these artifacts. This comparison shows that the problem lies in directional error within low-density, high-uncertainty regions: merely changing the step-size of the sampler does not help if the direction itself is unreliable. Curvature information is therefore essential for steering the trajectory away from such regions.

[1] Sumukh K. Aithal, Pratyush Maini, Zachary Lipton, and J. Zico Kolter. Understanding hallucinations in diffusion models through mode interpolation. Neurips 2024.

---

Q3: Human Evaluation Trustworthiness

As shown in Appendix B.6, our rating rubric is detailed.

Face images (AFHQv2, FFHQ) hallucinations are identified and annotated based on human perception, common sense, and visual plausibility. Hand images (11k-hands) are each assigned to one of seven categories: normal; extra fingers; missing fingers; incorrect finger structure; abnormal palm; multiple hands; unrecognizable/implausible.

All images were labeled independently by two human raters to ensure objectivity. Whenever their judgments diverged, the pair reviewed the cases together and discussed until they reached a consensus, producing a single, agreed-upon label for each image. This consensus process both minimized individual bias and improved the overall reliability of our ground-truth data.

To evaluate RODS-CAS’s impact relative to the baseline (EDM-Euler), annotators first identified hallucinations in the Euler-sampled outputs. They then compared each corrected image against its Euler counterpart, categorizing the result as “better,” “worse,” “same," or "unclear". This side-by-side comparison made it easy to see whether our method truly reduced artifacts, or introduced new ones. We will further update our Appendix B.6 with more details.

Finally, in the interest of transparency and reproducibility, we will make our complete annotation guidelines, raw labels, and final consensus labels publicly available. This will allow other researchers to verify our results and build on our work with full confidence in the human-evaluation process.

---

Q4: Selection of Hyperparameter

Thank you for the question. Our paper has provided simple, data-driven principles for both hyper-parameters, detailed as follows. We will study the automated hyperparameter selection in the future.

$\epsilon$ — detection threshold (Section 4.1, Appendix B.2)

• Role: controls sensitivity; lower ε is more sensitive to the score changing, i.e., uncertain steps.
• Guideline (line 205–207): “In high-stakes domains such as medical imaging, a lower $\epsilon$ may be preferred to prioritise sensitivity.”

$\rho$ — perturbation radius (Section 4.1)

• Role: sets the neighbourhood size over which we probe the score field.
• Choice: match $\rho$ to the characteristic feature scale of the dataset. If the surrounding region is trustworthy (e.g., low visual complexity), a larger $\rho$ is safe; if not, a smaller $\rho$ focuses detection on truly uncertain zones.

Dear Reviewer jSsH,

Thank you for taking the time to review our paper and providing your insightful feedback. We also appreciate you sharing the mandatory acknowledgement.

As the author–reviewer discussion period concludes in two days, we would be grateful if you could share your thoughts on our rebuttal at your earliest convenience. If you have any remaining questions or concerns, we’d be happy to engage in further discussion. If our responses have resolved your concerns, we would greatly appreciate your reconsideration of our score.

W1 & Q4: Trade‑off between hallucination mitigation and generation diversity

We fully agree that a mitigation technique should not come at the cost of collapsing the model’s creative space. Intuitively, RODS touches only a small subset of trajectories, so most images remain exactly the same. Visual evidence of this can be found in Appendix B.7 (Figures 13 and 14), while standard diversity metrics further confirm this observation.

Below we quantify and explain why RODS does not suppress diversity. In particular, we assess diversity across 1k samples using: Inception Variance [1], which quantifies variability in the latent representations, and DreamSim Diversity [2], which measures the variance of features extracted from a batch of generated images (batch size = 64). The experiment results show that RODS-CAS preserves generation diversity on par with the EDM-Euler method while effectively mitigating hallucinations, as detailed in the paper.

[1] Miao, Zichen, et al. "Training diffusion models towards diverse image generation with reinforcement learning." CVPR. 2024. [2] Domingo-Enrich, Carles, et al. "Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control." arXiv preprint 2024.

---

W2: Clarifying the assumptions behind the curvature proxy

We appreciate the reviewer’s comment on discussing the assumptions underlying our curvature-based detection. The key hypothesis is that hallucination correlates with model uncertainty [1]. As mentioned in our paper, "in well-behaved regions, this vector field changes smoothly. But in low-density areas, $v(x)$ can shift direction quickly—these are the regions where hallucinations are likely to occur" (line 467-471). By probing these rapid, spatial fluctuations, i.e., second‑order changes or “curvature”, we obtain a quantitative signal (our curvature index $H(x)$) for potential sampling failures.

In terms of the naturally sharp but valid features, such structures (e.g., object edges) are unrelated to the curvature index $H(x)$ used in our method. While such sharpness in image exhibits high image gradients $\nabla_x I(x)$, this differs fundamentally from the density-field gradient $v(x) = \nabla_x \log p_t(x)$, which reflects changes in data density. Our curvature index $H(x)$ measures variations in this score field—not in image intensity.

[1] Sumukh K. Aithal, et al. Understanding hallucinations in diffusion models through mode interpolation. Neurips 2024.

---

W3 & Q2: Detailing our qualitative survey and rating rubric

We thank the reviewer for pointing this out.

For hallucination labeling (Appendix B.6) :

-> Face images (AFHQv2, FFHQ) hallucinations are identified and annotated based on human perception, common sense, and visual plausibility. Hand images (11k-hands) are each assigned to one of seven categories: normal; extra fingers; missing fingers; incorrect finger structure; abnormal palm; multiple hands; unrecognizable/implausible.

Current use of “better”/“worse”/"same", and "unclear".

To make our evaluation protocol more precise, we conducted pairwise comparisons by human raters on anonymized image pairs (RODS vs. baseline). Unlike traditional binary hallucination labels, these judgments reflect relative image quality rather than the mere presence or absence of hallucinations. As a result, it provides a more nuanced and reliable assessment of generation fidelity. Furthermore, in the interest of transparency and reproducibility we will share our code, data, and labels in Github. More specifically, the detailed rubric is shown below:

• Better: The RODS-generated image is perceived as more faithful or realistic, often due to improved correction of visual anomalies (e.g., fixing a misplaced eye) without introducing new defects.
• Worse: The RODS-generated image is clearly degraded relative to the baseline, typically because it introduces new artifacts or hallucinated elements that were not present in the original.
• Same: No human-identifiable difference exists between the baseline and the RODS-generated image.
• Unclear: A change occurs, but it does not involve new hallucinations, and it is difficult to judge which version is better (e.g., slight pose changes or missing accessories like a ring).

---

W4 & Q3: Side-effects of corrections

We really appreciate the reviewer’s careful analysis and thoughtful comments—these are valuable and nuanced issues that touch on the core difficulty of hallucination correction. Due to the inherently ambiguous definition of hallucination, our correction strategy is not able to make fully “intended” edits. Rather, RODS introduces conservative updates at points identified as high-risk under the base model’s score landscape.

1. Imperfect correction: The effectiveness of correction is fundamentally limited by the base model itself. For instance, in cases like Row 3 (right), where the baseline model tends to denoise the fifth finger into blank space, our robust update may reintroduce it in a subtle but imperfect form (e.g., slightly discolored or ill-shaped), as we can only perturb within the model’s generative prior.

2. Semantic drift: On one hand, semantic drift is less relevant in our experiments, as there is no external prompt or class label whose meaning could shift. On the other hand, different diffusion timesteps correspond to different levels of abstraction. Among changes, edits made at early stages (high t) can result in semantic drift, while those at later stages (low t) typically affect finer details. Our current method does not constrain which timesteps are eligible for correction, as we found that some large-scale hallucinations require intervention at earlier stages. However, such early-stage perturbations may also affect high-level semantics, such as altering the orientation of a hand, as seen in Row 4 (right). If semantic consistency is a priority, one could optionally restrict updates to later timesteps to avoid undesired high-level changes.

3. Over-sanitization: Quantitative diversity remains almost the same: as shown in W1&Q4. Additionally, since most samples are left bit-wise unchanged (Fig 13-14), the model’s creative range is effectively preserved, addressing the over-sanitization concern. We will post all 1k results and the human label for public review.

We acknowledge that achieving more targeted and controllable corrections would require more precise hallucination localization and timestep-aware update scheduling. We view this as a promising direction for future work.

---

W5: More discussion on limitation

Thank you for highlighting these important considerations. Here is how we will address them in the revised manuscript:

1. Architectural dependence (VE formulation)
   Our RODS algorithm can be directly implemented within the VE-ODE framework, which benefits from the equivalence between VE-based sampling and the continuation method under the usual parameterization (Section 3). We will clarify this dependency and note that extending RODS to other sampling schemes (e.g., VP or latent-space diffusion) is conceptually straightforward but remains to be validated experimentally.

2. Dataset scope limitations
   Currently, we have tested RODS on well-established benchmarks (AFHQv2, FFHQ, 11k-hands), where hallucinations can be easily observed without domain expertise. We acknowledge that in highly diverse or specialized domains, i.e., medical imaging, the definition of “hallucination” must be grounded in expert judgment. We are collaborating with domain practitioners to evaluate RODS under these conditions.

---

W6: Relevant work suggestion

Thank you for pointing out this relevant work. While it takes a different approach (adaptive attention modulation) compared to our trajectory-level geometric regularization, we agree that it addresses a related goal. As it was posted on arXiv in February 2025 and had no available code at the time of our submission, a direct experimental comparison was not feasible. We will gladly include a citation and discuss its relation to our work in the revised section.

---

Q1: Selection of epsilon

In our current experiments, we observed that the choice of $\epsilon$ is influenced by both the dataset characteristics and the desired sensitivity level. For instance, 11k-hands features clean backgrounds and fine-grained hand details, so a smaller $\epsilon$ helps detect subtle hallucinations in localized regions. In contrast, face datasets (AFHQv2, FFHQ) often involve more complex backgrounds, where a slightly larger $\epsilon$ is preferred to avoid overreacting to natural variation in curvature. More discussion about the epsilon selection has been presented in Section 4.1 (line 205-207) and Appendix B.2. Given a desired sensitivity level, we believe it is possible to automatically choose $\epsilon$ (there are similar approaches in traditional literature), but we consider it as a future research direction.

Dear Reviewer GnZW,

Thank you once again for your time and valuable feedback on our paper. We have carefully considered your detailed comments and addressed your concerns in our rebuttal above.

If you have any further questions or would like us to clarify any points, we would be more than happy to provide additional details or engage in further discussion. If you feel our response addresses your concerns, we would appreciate your reconsideration of our score.

I appreciate the authors' efforts to respond to the raised concerns, where most of my concerns have been addressed. However, I have a few follow-up questions for the authors.

---

W1 & Q4: Trade‑off between hallucination mitigation and generation diversity:

Including DreamSim and Inception Variance helps in quantifying the generative capability, where we see that the proposed model reports slightly weaker metrics compared to the other models except for DreamSim in 11K Hands --- but yet may be considered at par as the authors have stated. In addition to the reported comparisons, I believe the authors need to include the metrics for the default diffusion models (w/o any mitigation strategy) to set the baseline for the generative capability.

---

W4 & Q3: Side-effects of corrections

I agree with the stance of the authors, where mitigating hallucinations at early denoising steps could lead to unwarranted/uncontrollable side effects such as semantic drift. While it is true that all the results shown are in unconditional settings, consider the practical aspect of using RODS with the generation.

For example, if a user simply wants a bunch of random hands, they could either filter out the hallucinations (using a detector as proposed in [1]) or alternatively use RODS to create a pool of hand images (which would be similar to the filtered sample set). I would appreciate if the authors could explain why would once choose one over the other. Additionally, while I don't undermine the authors' work by any means, I believe that correcting a hallucination artifact while preserving the other semantics of the image has a much higher practicality. For example a user would be interested in correcting a hallucination of a particular image because the rest of the semantics are appealing/of use, else would perhaps choose to discard (given the model is capable of generating a large corpus).

---

W5: More discussion on limitation | 2. Dataset scope limitations

I agree that a domain such as "medical imaging", could be handled through expert judgement. However, consider a diffusion model that is trained on a much diverse corpus (e.g. Stable Diffusion trained on LAION5B), in such cases I would like to know the authors take on the scalability of the proposed algorithm. From my understanding, the RODS is limited to diffusion models trained on datasets with one/few/less-diverse classes.

---

Finally, I would like the authors to list the planned amendments to the final manuscript. The authors need not go into details, a concise list would suffice.

[1] Sumukh K. Aithal, et al. Understanding hallucinations in diffusion models through mode interpolation. Neurips 2024.