Regularization by Texts for Latent Diffusion Inverse Solvers

As I mentioned in my answers above I appreciate the effort in providing more results and clarifications. Many aspects have been indeed clarified and the experiments added do provide a bit more evidence, although in my opinion, such evidence should be stronger. I do like the work but the experimental evaluation could and should be stronger. In light of this I am going to updated my scores and overall recommendation.

Thank you for your reply and for considering my suggestions. I would like to start by highlighting that despite the initial rating I like this work. My comments are only aiming to improving it further to the point that it shows stronger scientific rigor. I hope my feedback can help towards making the paper stronger, I am not trying to discredit this work.

Re W2 & Q2 Thanks for providing more quantitative results and a comparison with the highly related P2L baseline. While I was expecting something like P2L Table 2 and 3 (i.e., FFHQ and ImageNet on super resolution, motion deblur, gauss deblur, and inpainting). That said, the new results can strengthened a bit the original submission. The weakness of the new results is that they represent only 3 classes (persons, cats and dogs) and it is unfortunate they do not include inpainting. Following the experimental settings of P2L would have shown results on the whole variability of ImageNet which would have been even more convincing.

Re W3 Thank you for adding a few more classes to your experiments. Similarly to my sentiment above: I appreciate the effort you placed into performing more experiments but this is not exactly what I was asking for. I was asking for quantitative evaluation on full datasets. In my opinion to support the claims of this section one would need to show results on the full dataset translated to at least 1 other class. Without this a reader will always be left asking if these were hand picked examples or if the algorithm really generalizes.

Statements like TReg is comparable or superior compared to state-of-the-art image editing algorithms, highlighting the efficacy of text-guidance facilitated by TReg, or TReg effectively integrates the given text prompt while preserving robust data consistency, thereby validating its efficacy, would carry a stronger message if they were supported by strong evidence rather than evaluating the algorithm on a handful of classes.

Minor: Since you use the clip similarity score I would consider adding for every class you the average clip similarity of the original class (from validation set), just to have a reference point (sine I assume even that similarity will not be 1.0).

Re W1 and W4 I appreciate that some experiments can be meant simply as a proof of concepts, however, I hope you can also appreciate that showing proof of concepts, while encouraging, is not sufficient to make any definitive claim.

W9. (continue)

• “Here, ζ,γ are empirically chosen to satisfy¯αt−1 = ζ/(ζ+ γ).” Please provide an explanation why satisfying this equation is important.

  $\zeta$ and $\gamma$ are hyperparameters for the proximal optimization problem, chosen to satisfy the second equality in (15). Specifically, we adjust the interpolation coefficient to prioritize $\hat{z}_0(y)$ during the final phase of reverse sampling, ensuring fine-grained refinement for improved data consistency. Empirically, this strategy demonstrates robust performance, as described in the revised manuscript. This is potentially attributed to the coarse-to-fine detail refinement inherent to reverse diffusion sampling.

• In the experimental section consider breaking down the baselines for each experimental task. Currently they are presented all at once at the beginning irrespective of the tasks and I find this to be too much information to process and too far from where it is useful.

  About baselines, we wrote their features in lines 349-361.

• The authors used the prompt “A photography of a ….”. This might not be grammatically correct. Photography is the art of creating photos or the process of creating photos. A better prompt would be “A photo of a…” or “A photograph of a …”. I would be curious to know if this changes the results.

  Negligible. Please see the response to Q7.

• Details in Figure 1 caption. Tweedie’s formula should be cited. From (10) to (13) the terms are swapped. Consider clarifying the diffusion model is pre-trained. Rephrase suggestions.

  Fixed.

Q3. In figure 2 authors show that thanks to the adaptive negation strategy the generated image is less noisy. It is unclear however why this would be the case. Could it be that without adaptive negation one could reach a similar results with more iterations?

A. Because the prior given by text description corresponds to clean images, adaptive negation is also effective to reduce blurry effects which do not exist in clean images. Even though it is possible to reach similar results with more iterations instead of adaptive negation, it is beneficial to use the adaptive negation as it may accelerate problem solving.

Q4. Due to (18) does it mean that the VAE needs to have the same dimensionality of CLIP embeddings?

A. We guess that the reviewer is pointing out the equation (19). As noted in line 282, $\mathcal{T}_{img}$ denotes the CLIP image encoder, which maps the image to CLIP embedding space. Therefore, VAE does not need to have the same dimensionality of CLIP embeddings.

Q5. “The proposed solver without adaptive negation tends to achieve higher PSNR than with adaptive negation, since it obtains blurry images”. Could the Structural similarity index measure (SSIM) be a better metric in this case?

A We observed that SSIM shows the same tendency as the PSNR. Thus, we reported FID which indicates the distance of two distributions.

Q6. For the phase retrieval experiments: the error in the LDPS reconstruction seem to go beyond the ambiguity of symmetries… Do the authors have any intuition about why the quality of LDPS reconstruction is so low compared to TReg?

A. Because we are conducting all experiments with 200 NFEs, while both DPS and PSLD have used 1000 NFEs in their paper. Reducing the number of timesteps can cause insufficient update of latent code and results in lower quality of reconstructions. In contrast, TReg can effectively solve the problem within 200 NFEs which is beneficial in its efficiency.

Q7. How sensitive is this frameworks to slight variation of the text? This would be an interesting discussion of the paper. The authors use “A photography of …” how much do results change if instead they used “A photo of xxx” or simply “xxx”?

A We appreciate the reviewer’s constructive feedback. While the overall performance is not highly sensitive to grammatical details, we observed marginal improvements when adjusting the prompt as kindly suggested by the reviewer (Table 4). This observation suggests that TReg can potentially benefit from carefully crafted prompts.

[Table 4] Sensitivity to text prompt. Evaluated on Deblurring task with AFHQ-cat.

W6 & Q1. For the alternate directions method there should be some additional information about convergence. Being an iterative method one would expect to see some analysis about it but I did not find any information.

A. We thank the reviewer for the constructive comment. Our approach optimizes the full objective in an alternating manner, where the data consistency is optimized in pixel space and the text-conditional regularization is mainly applied in latent space. A direct theoretical convergence analysis is challenging due to the complexity arising from the transition between pixel and latent spaces. However, in the Appendix B of the revised manuscript, we provide an informative empirical analysis by evaluating the progression of two objectives during the reverse sampling process: data consistency in pixel space and the denoising score matching loss (i.e. score distillation sampling loss) conditioned on a given text prompt. Specifically, we plot $\| y - A(\hat x_{0|t}) \|^2$ for data consistency, which measures how well the solution aligns with the given measurement. Also, we measure $| \epsilon - \epsilon_\theta (x_t, t, c) |^2$, a text-conditional diffusion model objective which evaluates how well solutions align with the clean data manifold and the corresponding text condition c. We present results for two scenarios: when the text prompt matches the ground-truth label and when it differs. For both cases, we plot the mean and standard deviation of these objectives, averaged over 100 samples. As shown in Figure 8 in the appendix of the revised manuscript, both objectives decrease significantly during the reverse sampling process, demonstrating that the TReg empirically promotes convergence. Notably, the data consistency of TReg improves particularly in the later sampling phase, which is in line with the interpolative constant setting in eq (13) of the revised paper. Please refer to W9-4 response for more detail.

W7. The work of Mokady et al also shares the idea of adaptive negation (but with different application and implementation) so for Inpainint experiments (which are suitable for Mokady) this would also be a useful baseline.

A. NTI is fundamentally different from TReg although they use the null-text embedding as a parameter for updates. NTI focuses on editing tasks, where the null-text embedding is updated to ensure that the reverse sampling trajectory does not deviate significantly from the pivot trajectory generated by DDIM inversion with a conditioned score function (i.e. CFG guidance scale of 1.0). This null-text is designed to preserve the contents of the source image while incorporating the editing text prompt. Therefore, it does not involve suppressing complementary concepts, as achieved by adaptive negation. More importantly, TReg does not incorporate the inversion process and also NTI does not incorporate data consistency problems.

Writing We really appreciate the reviewer for carefully reading our manuscript and providing comments to enhance the readability. We fixed it by considering feedback from the reviewer. Also, we would like to clarify some unclear phrases that the reviewer mentioned point-by-point in below.

W8. Background session could be summarized and moved to the appendix since these are all known concepts.

A. Thanks for the suggestion. We moved section 2.2 (classifier-free-guidance) to the appendix. However, section 2.1 and 2.3 are crucial as they provide the context of this study which is important for one who is not familiar with diffusion-based inverse problem solving, so we maintain them in the background.

W9. The additional space gained should be used to provide more details about the proposed method.

• authors should mention about update range in the main paper

  We would like to gently clarify that it is common to update the latent code “every” iterations during the reverse process in diffusion-based inverse solvers, including DDRM, DPS, PiGDM, PSLD, and even for P2L. However, we just skip some steps for computational efficiency. That being said, we clarify the details of the update range in the Appendix because it is optional.

• Consider moving the inpaiting results (with comparison with Mokady) in the main paper.

  Due to page limitations, we provide a representative example in Fig. 1, and refer readers to the appendix for additional results per reviewer’s request. For the comparison with Mokady, please refer to our response to W7.

• Term A(x) in Eq (13) and (14): it is not clear the role it plays in practice when solving a specific problem.

  We would like to gently remind the reviewer that it is a conventional term in inverse problems. However, we have revised the phrase to "forward measurement operator" for clarity.

W5&Q2. This work shares a lot of similarities with P2L (Chung et al.) ... In the intro when talking about P2L the authors say “However, this method primarily focuses on data consistency and lacks robust alignment with textual prompts.” It would be great to support this statement with quantitative results.

A. We kindly remind the reviewer that P2L is designed to update text prompts to enhance data consistency. While P2L can initialize the text prompt, this functionality is solely aimed at achieving better consistency with the measurement and does not facilitate finding a solution that aligns with the given text prompt.

Please refer to Appendix C of revised manuscrip for more discussions on P2L. As mentioned in lines 78-82, we highlight that P2L is orthogonal to our approach, despite their apparent similarities. Specifically, P2L merely treats the null-text prompt as an additional parameter to improve the data consistency, optimizing it as:

$\mathcal{C}^* = \arg \min_{\mathcal{C}} \| \mathbf{y} - \mathcal{A}(\mathcal{D} (\hat{\mathbf{z}}_{0|t}(\mathcal{C}))) \|^2$ (Eq. 12 in P2L).

Unlike TReg, P2L does not guide outputs toward a specific mode aligned with semantic linguistic conditions. It lacks support for Classifier-Free Guidance (CFG) and relies solely on conditional scores. While both methods leverage prompt embeddings, they do so in fundamentally different ways: P2L focuses on data consistency, whereas TReg reduces the solution space by incorporating rich perceptual estimates of noisy measurements in the form of linguistic descriptions.

To adapt P2L as a text-based baseline, we assume an ideal scenario where the ground-truth class is provided for the noisy measurement. Please refer to the general comments for the results on the full validation set of FFHQ and AFHQ. The text embedding is optimized by using the Adam optimizer (learning rate 1e-4) with K=1. We perform reverse sampling with 200 timesteps, consistent with other baselines. That said, we note that P2L requires twice the neural function evaluations (NFEs) of TReg, as it recomputes Tweedie estimates after updating the text prompt. Further runtime analysis is included in the Appendix. Moreover, to fully address reviewer concerns, we also evaluate P2L in a more practical scenario where the estimated (non-ground-truth) class is provided as text, simulating real-world user text cues. As shown in Table 3 of the response and Figure 5 in the revised manuscript , P2L and other baselines fail to align outputs with the provided text, as they do not fundamentally encourage alignment with the given text. Please refer to W6 & Q1 for more comparative analysis on the sampling process.

[Table 3]. Food - main experiment, comparison with P2L / NFE 200

W1. Ambiguity reduction: it is unclear if the quantitative evaluation in Figure 3 is an average over a dataset or simply the results of the image shown (bear). It is also unclear from which source this image comes from.

A. We would like to clarify that the variance plot in Figure 3 is drawn by repeating reconstruction 10 times with a single measurement, where the true image is sampled from the ImageNet validation set. We added this detail in the revised manuscript. As our goal was to provide a proof-of-concept demonstrating the benefits of text regularization in reducing ambiguity, we believe this level of detail is sufficient. In fact, the remaining part of the experiments demonstrates the ambiguity reduced by TReg. We added the detail in the revised manuscript.

W2 & Q2. Accuracy of obtained solution: use true class as c. Results in Table 1 and Table 2 show quantitative measurements (great) for two tasks (super resolution and de-blurring) but only on two classes “Ice-cream” and “fired rice” rather than averaging across full datasets.

A. Thank you for the valuable comment. In Table 1 of the general comment, we evaluated TReg on the full validation sets of FFHQ, AFHQ-cat, and AFHQ-dog—widely used benchmarks for inverse problem solvers—using the true class as a text condition per Reviewer’s request. These datasets provide representative and measurable results compared to full Food-101 validation, which includes over 100 classes. To strengthen the comparison, we also included P2L as a baseline (Please refer to W5 & Q2 and Appendix C for more discussions on P2L). Table 1 demonstrates that TReg consistently outperforms other baselines across all inverse problems with high ambiguity. This advance stems from the given text conditions which effectively narrow down the solution space. These results have been added to Table 5 in the revised manuscript.

W3. Accuracy of obtained solution: use different class as c. Results shown in Table 3 are for only two classes using two different C labels (fried rice to spaghetti, and Ice Cream to Macaroons). ...the quantitative evaluation should be reported on a larger datasets (with mean and standard deviations).

A. Thank you for the thoughtful and valuable feedback. As requested, we have compared TReg and the baselines on the full FFHQ and AFHQ validation sets, widely used benchmarks for inverse problems, with true class text conditioning (Table 1 in general response). Additionally, we conducted more experiments on Food-101 subsets (e.g., “risotto → french fries” and “gnocchi → spaghetti”), representing practical scenarios where perceptually estimated (non-ground-truth) classes are provided as text cues (Table 2 in general response). We believe these results are both representative and measurable, especially compared to the full Food-101 validation set of over 100 classes. Given that most diffusion-based inverse problem solvers typically use 2-4 datasets, we would like to gently assure the reviewer that our evaluation is sufficient to demonstrate the method's effectiveness. Please see the general comments and responses to W5 and Q2 for further discussion.

We would like to gently clarify that our evaluation pipeline is carefully designed to assess both (a) data consistency and (b) textual alignment simultaneously. This differs from image editing tasks, which often prioritize preserving coarse level of spatial layout over strict alignment with measurements. In our context, reconstructed solutions must align with noisy measurements while remaining perceptually plausible for human users. Evaluating whether measurements can be “translated” into clean solutions for arbitrary classes is beyond the scope of our work and would require measurement-specific benchmarks. We are excited to see future works in this direction.

W4. Results on non-linear inverse problems. These results are only qualitative shown on 4 different images (Figure 6).

A. We would like to assure the reviewer that they are proof-of-the concept experiments to show the TReg also works for non-linear cases, which is different from two main inverse problems in our paper. In the final version of the paper, we will try our best to include quantitative results with more extensive experiments.

Thanks for the prompt response. However, the reviewer’s comment is a bit confusing as in the previous round the reviewer said “ In my opinion to support the claims of this section one would need to show results on the full dataset translated to at least 1 other class”.

If the reviewer is asking why we did not conduct an experiment of solving the inverse problem using true class as text description for ImageNet, we would like to assure the reviewer that we will ensure to include the results for ImageNet in the final version of the paper. Due to the time constraints of the rebuttal period, we were unable to include comparisons with all baselines for all tasks. But we tried our best to conduct additional experiments on box inpainting problems to address the reviewer's comments.

On the other hand, if the reviewer is asking the reason for using text description that differs from the original class, the reviewer is kindly reminded that the primary focus of this paper is not solving conventional inverse problems. Instead, our aim is to demonstrate the feasibility of reconstructing solutions using text-based regularization. Consequently, we prioritized and spent our time validating the core contributions of our paper, as reflected in Table 6 of the main response and Figure 16 in Appendix H.

Hope this addresses your question and please feel free to contact us if there are more things to clarify.

W1. Line 071: In my view the connection to the human brain is quite weak

A. We agree that Line 71 does not provide core insight for the proposed method, so we excluded it in the revised manuscript. Thanks for carefully reading the manuscript.

W2. I would like to see additional, none cherry picked outputs. ...specifically interested in image inpainting results, which nearly always result in boundary artifacts.

A. We added uncurated samples for the inpainting problem in the Figure 17 of the Appendix of revised manuscript. Although there are few cases with remaining boundary artifacts for large box inpainting (Figure 17 bottom), TReg successfully mitigated boundary artifact issue.

In case of inpainting, boundary artifacts occur due to inconsistency between in-mask and out-mask regions. To address this effectively, information of in- and out-mask regions should be mixed via algorithms like DPS [1, 2]. As noted in the section 3.2, TReg also requires steps of LDPS to elaborate this consistency. Nevertheless, TReg provides controllability on how to fill-out the masked region according to text description. On the other hand, a naive combination of PSLD and CFG did not work as shown in Figure 16 in revised Appendix.

[1] Diffusion Posterior Sampling for General Noisy Inverse Problems

[2] Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models

W3. I would like to see experiments in the presence of measurement noise, for example with JPEG compression artifacts after gaussian blurring.

A. All the experiments are done with the presence of Gaussian noise to the measurement (line 304). Although the TReg shows the robustness to the measurement noise, we additionally tested another type of measurement noise: JPEG compression artifact. Specifically, we use the forward operator implemented by DDRM. For JPEG compression, we vary the quality factor (QF) at 50, 30, and 10, where a higher quality factor indicates less compression.

As shown in Figure 11 in Appendix, TReg demonstrates robustness up to Q30, providing nearly identical reconstructions. At Q10, however, significant noise is introduced into the measurement, resulting in degraded reconstructions. For such extreme JPEG artifacts, incorporating JPEG compression into the forward operation could be considered.

W1. Lack of overall results over whole dataset

A. Thanks for the constructive comment. As requested, we compare TReg and the baselines on the full FFHQ and AFHQ validation sets–widely used benchmarks for inverse problems–with true class text conditioning in Table 1 of the general comment. To further simulate the unknown class scenario (where a perceptually plausible class is inferred and provided as text conditioning), we included additional experiments with more class subsets from the Food-101 dataset (“risotto -> french fries“ and “gnocchi -> spaghetti“ ). In both cases, TReg demonstrates its superior performance in terms of image fidelity and textual alignment. For detailed results, please see the general comment.

W2. In both table 1 and table 2, PSNR scores are higher without adaptive negation, do authors have any analysis or intuition about this results?

A. As mentioned in the main paper (line 415-416) , PSNR tends to be higher when the image is blurrier, regardless of image fidelity. This is evident in Figure 2(b). Therefore, we reported a perceptual metric, FID, in addition to the PSNR.

W3. (this might not be a weakness, just naturally curious) in the experiments, the text regularization is only tested with class name, how about the results using some natural captions (such as generated with BLIP or GPT)?

A. Thank you for the insightful question. We believe natural captions generated by LLMs are compatible with TReg, offering the potential to enrich text conditions and automate the reconstruction process with reduced human intervention. That said, it is more crucial to investigate how to improve VLMs’ accuracy in inferring classes from noisy measurements. Moreover, Stable-Diffusion v1.5 involves CLIP as text encoder which limits the maximum context length and its compatibility with other advanced language models. We believe extending TReg with more powerful text encoders would be an interesting future work.