Learning Single Index Models with Diffusion Priors

Thanks for your recognition of this paper and the helpful comments and suggestions. Our responses to the main concerns are as follows. All citations refer to the reference list in the main document.

(To strengthen the paper, it would be beneficial to include a version of Theorem 2 for the SIM-DMS algorithm (28), if feasible. For now, only the SIM-DMIS algorithm is endowed with the theoretical guarantees. Besides, Theorem 2 does not bear entirely on the SIM-DMIS estimator. Would it be feasible to combine (24) and (29) to reach an end-to-end guarantee?) We are grateful to the reviewer for the valuable suggestions. In the revised paper, we will include a version of Theorem 2 for SIM-DMS. Eq. (29) within the statement of Theorem 2 essentially says that for any sample drawn from the marginal distribution $q_t$, when we perform the inversion process from time $t$ to $T$, followed by the sampling process from time $T$ to $\epsilon$, the resulting vector will be reconstructed to closely approximate the corresponding ground-truth data from $q_0$ (and lie on the same ODE trajectory as the original sample). Additionally, Eq. (24) (along with Eq. (25)) essentially indicates that (the scaled version of) $\frac{\mathbf{A}^T\mathbf{y}}{m}$ approximately follows the marginal distribution $q_{t^*}$ for some $t^*$. If we make the (relatively strong) assumption that (the scaled version of) $\frac{\mathbf{A}^T\mathbf{y}}{m}$ precisely follows $q_{t^*}$ for some $t^*$, then combining Eqs. (24) and (29) provides an end-to-end guarantee. We leave the end-to-end guarantee in the case where Eq. (25) only approximately holds to future study.

(As a minor comment, the related work section could be further clarified, notably the last paragraph of page 2. Currently, it is unclear which work address linear or non-linear settings, and in which works the link function is (un)known.) Among the methods described in the last paragraph of page 2, MCG (Chung et al., 2022b) and $\Pi$GDM (Song et al., 2023) are mainly designed for the linear setting. Additionally, $\Pi$GDM can be extended to certain nonlinear settings (where the link function is known) by leveraging a combination of pseudoinverse operations and nonlinear transformations. DPS (Chung et al., 2023b) and DAPS (Zhang et al., 2024) are applicable in nonlinear settings as well, also primarily under the assumption that the link function is known. We will clarify these in the revised version.

Thanks for your helpful comments and suggestions. Our responses to the main concerns are as follows. All citations refer to the reference list in the main document.

(It is strange that DPS and DAPS with knowledge of the link function (i.e., DPS-N and DAPS-N) perform worse than they do without knowledge of the link function (i.e., DPS-L and DAPS-L).) In 1-bit measurements, even in the noiseless scenario, the link function $f(x) = \mathrm{sign}(x)$ is non-differentiable at $x = 0$ (though as mentioned in Footnote 9, PyTorch can still enforce automatic differentiation). The differentiability of $f$ matters significantly. As mentioned in Section 1.1, under the strong differentiability assumption on the link function, (Wei et al., 2019) can handle general non-Gaussian sensing vectors. However, most SIMs research assumes Gaussian sensing vectors, like the seminal work (Plan & Vershynin, 2016) and many follow-up studies such as (Liu & Liu, 2022). Without differentiability, extending these to non-Gaussian cases is difficult.

The non-differentiability also poses challenges for DPS-N and DAPS-N as they rely on $f$ in gradient based updates. This can lead to inaccurate gradients and ultimately resulting in subpar performance. (The DAPS supplementary material suggests using Metropolis Hasting for non-differentiable forward operators, but it is also mentioned to have inferior performance and low efficiency, with results only in the supplementary material.) In contrast, DPS-L and DAPS-L do not utilize $f$, thus allowing for relatively better performance (note that as demonstrated in the work (Plan & Vershynin, 2016), SIMs can be transformed into linear measurement models with unconventional noises).

(There is little intuitive explanation of Theorem 2 (I could not find a Theorem 2, by the way). And there is no discussion about whether this would provide samples from a Bayesian posterior.) Theorem 2 essentially says that for any sample drawn from the marginal distribution $q_t$, when we perform the inversion process from time $t$ to $T$, followed by the sampling process from time $T$ to $\epsilon$, the resulting vector will be reconstructed to closely approximate the corresponding ground-truth data from $q_0$ (and lie on the same ODE trajectory as the original sample). In the revised version, we will incorporate this intuitive explanation in the paragraph preceding Theorem 2. We are unsure what the reviewer means by “I could not find a Theorem 2”. We guess that the reviewer might be referring to the proof of Theorem 2, which is available in Appendix B.1.

The approaches presented in our work are not directly related to sampling from a Bayesian posterior. While it would be interesting to discuss whether our approaches could yield samples from a Bayesian posterior, we believe that such an exploration is beyond the scope of the current work.

(The authors should include more intuitive explanations. For example, what exactly are the properties in Eqs. 20 and 21 saying? What is the intuition behind Eq. 25 and why it’s different from Eq. 22? Also, the background on diffusion models is too dense and contains a lot of unnecessary information.) The condition in Eq. (20) is a classic and crucial condition for SIMs. For example, it is (albeit implicitly) assumed in the seminal work (Plan & Vershynin, 2016) and in subsequent research that builds upon it. If this condition fails to hold, specifically when $\mu = 0$, the recovery of $\mu \mathbf{x}^*$ as in (Plan & Vershynin, 2016) and in our Eq. (24) becomes meaningless. We follow (Liu & Liu, 2022) to assume the condition in Eq. (21), which generalizes the assumption that $f(\mathbf{a}^T\mathbf{x}^*)$ is sub-Gaussian (which is satisfied by quantized measurement models), and accommodates more general nonlinear measurement models, such as cubic measurements with $f(x)=x^3$ and their noisy counterparts.

The intuition underlying Eq. (25) is that (the scaled version of) $\frac{\mathbf{A}^T\mathbf{y}}{m}$ is approximately the ground-truth signal $\mathbf{x}^*$ (drawn from the target data distribution $q_0$) with an added zero-mean Gaussian noise component. Then, considering the forward process of diffusion models, which gradually adds Gaussian noise to the ground-truth data, performing a full inversion (from time $\epsilon$ to $T$) and then a full sampling from time $T$ to $\epsilon$ as in Equation (22) is inappropriate. As mentioned in the paragraph below Eq. (22), this approach implicitly (and wrongly) assumes that $\frac{\mathbf{A}^T\mathbf{y}}{m}$ approximately follows the target data distribution $q_0$ (without taking additive zero-mean Gaussian noise into account).

In the revised version, we will incorporate the above intuitive explanations. Additionally, we will streamline the background on diffusion models, focusing solely on the equations and concepts that are fundamental to understanding the core ideas presented in this paper.

Thanks for your recognition of this paper and the helpful comments and suggestions. Our responses to the main concerns are as follows.

(FID is widely recognized as a key evaluation metric in diffusion models. Why wasn’t it employed here?) We carry out additional experiments to report the FID. The results are presented in Table B1 below. Given the time constraint during the rebuttal period, and following the work for DAPS (Zhang et al., 2024), we compute the FID using a set of 100 validation images. The FID results for QCS-SGM, DPS, and DAPS are not included at this stage as their calculation is relatively time-consuming. However, in the revised version, we will incorporate the FID results for all methods.

(To ensure a fair and comprehensive comparison, I suggest conducting additional experiments, specifically comparing SIM-DMIS with 50 NFE and SIM-DMS with 150 NFE.) Thanks for the insightful comment. We have conducted simple ablation studies for SIM-DMS and SIM-DMIS on the CIFAR-10 dataset (mentioned in Footnote 10, and detailed in Appendices F and G), and the results show that further increasing the NFE for SIM-DMS does not lead to substantial performance enhancements. We also perform the suggested experiments on the FFHQ dataset, and summarize the additional results in the following table. The results also indicate that under the same NFEs, SIM-DMIS consistently outperforms SIM-DMS across all metrics.

Table B1: Quantitative comparison of SIM-DMS and SIM-DMIS on the FFHQ dataset.

(It would be beneficial to include additional comparisons highlighting the reconstruction speed of the proposed method relative to existing approaches.) We conduct additional experiments to measure the reconstruction speed of our method, and the results are presented in the table below. The reported inference time refers to the average reconstruction time for 10 validation images from the FFHQ dataset. All of these experiments are executed on a single NVIDIA GeForce RTX 4090 GPU. The results indicate that SIM-DMS and SIM-DMIS exhibit significantly faster reconstructions when compared to the competing methods (we do not compare with QCS-SGM since it is very time-consuming). We will include the comparisons with respect to reconstruction speed in the revised version.

Table B2: Comparisons for reconstruction speed on the FFHQ dataset.

(Some terms in the references need to maintain consistency with the original papers, particularly regarding capitalization.) In our revised version, we will carefully check all the references and correct all the inconsistent cases to ensure accurate citation formatting.

We thank the reviewer for the feedback. We are pleased that the claims in our submission have been recognized as "supported by evidence" and our experiments have been recognized as "sound". Regarding your major concern about the practical relevance of Theorem 2, our responses are as follows:

• For practical relevance, the parameter $h_{\max}=\max_{i \in [N]} \big(\lambda_{t_i}-\lambda_{t_{i-1}}\big)$ in Eq. (29) plays an important role. When the number of sampling/inversion steps is large, for instance, on the order of $10^2$ or $10^3$, $h_{\max}$ is approximately $10^{-2}$ or $10^{-3}$ (if using typical uniform $\lambda$ steps). Therefore, when the number of sampling/inversion steps is sufficiently large, in the upper bound $C h \sqrt{n}$ illustrated by the reviewer, the value of $h$ will be much smaller than $1$ (e.g., $h = 0.001$; note that $C$ is a positive constant of order $\Theta(1)$). This makes the upper bound practically meaningful when image pixels are bounded within the range $[0,1]$.
• The proof of Theorem 2 (see Appendix B.1) is built upon the proof for Theorem 3.2 in the popular work for DPM-Solver (Lu et al., 2022a) (note that the upper bound in (Lu et al., 2022a, Theorem 3.2) is for each individual pixel, and thus their bound does not have the $\sqrt{n}$ factor), which is in turn based on classic analyses of local truncation error for numerical ODE solvers, such as those presented in Section 5.6 of (Burden & Faires, 2005). And our Theorem 2 bears similar practical relevance to the theoretical findings presented in these works.

Richard L. Burden and J. Douglas Faires, Numerical Analysis, 8th edition, Thomson/Brooks/Cole, 2005.