Demystifying Spectral Feature Learning for Instrumental Variable Regression

We thank the reviewer for their careful reading and insightful comments on our paper. We have addressed the reviewer concerns and questions below, and we believe our proposed revisions strengthen the manuscript.

We would be grateful if the reviewer might consider increasing their score in case they are satisfied with the proposed changes.

• Weakness 1 & Question 1. Detailed comparative analysis in literature review. We agree that given the large amount of NPIV papers, a more explicit discussion of the similarities, key differences, and unique theoretical advantages or limitations of our approach compared to these relevant methods would strengthen the paper's positioning within the literature. We will revise the related work in the camera-ready version to include this expanded theoretical comparative analysis and the missing citation to "Dual Instrumental Variable Regression" and "Fast Instrument Learning with Faster Rates".

• Weakness 3 & Question 2. Experimental comparisons with state-of-the-art methods. It is important to clarify that our paper primarily focuses on the theoretical characterization of spectral features and their performance regimes (good, bad, ugly) for NPIV. The spectral features framework, which our paper analyzes theoretically, has already been empirically benchmarked extensively against a range of other state-of-the-art methods by [1] (including KIV, DFIV, DeepGMM and Dual Embedding). That study provides comprehensive empirical evidence of its practical performance against established benchmarks. Therefore, a full-scale empirical comparison was not the main objective of the present theoretical contribution. We will add a sentence to the experimental section of the main paper to explicitly direct readers to this relevant work and its comprehensive benchmarks.

• Weaknesses 2 & 4. Experimental generalizability to complex, high-dimensional data. We agree that our experimental section could be expanded to better illustrate generalizability from synthetic data to complex, high-dimensional, real-world settings where neural networks are typically employed.

  To address this, our paper now includes empirical results on the dSprites dataset. This is a well-established, high-dimensional dataset for NPIV, serving as a crucial example of a complex scenario where our method (which uses neural networks for feature learning) demonstrates strong applicability and scalability. We refer to our answer to W1 of Reviewer m9NX for further details. Furthermore, the spectral contrastive learning approach has previously achieved state-of-the-art results on challenging Demand Design datasets [1], further proving its efficacy on complex real-world data.

  While our synthetic experiments use trigonometric functions to clearly isolate theoretical effects, we believe our new results on dSprites and the performance on Demand Design in prior work provide strong evidence for the generalizability of our findings to more complex settings.

• Weakness 5. Missing conclusion section. We will use the extra page given for the camera ready version to add a concluding discussion and perspective on future works. We acknowledge that key limitations of our submitted work included the absence of a strategy to empirically quantify spectral alignment and decay for identifying the "good, bad, or ugly" scenarios, and the lack of a precise theoretical characterization of the error term $\varepsilon$ in Proposition 3. We have addressed the former through a newly developed empirical strategy for spectral alignment, as elaborated in our response to W1 for Reviewer m9NX. We also recognize the latter as an important and open problem for future research, which we briefly discussed on page 8 of our manuscript and further elaborate on in our response to W3 for Reviewer m9NX. We will explicitly discuss these limitations in the conclusion.

[1] Haotian Sun, Antoine Moulin, Tongzheng Ren, Arthur Gretton, Bo Dai "Spectral Representation for Causal Estimation with Hidden Confounders", AISTATS, 2025.

We thank the reviewer for their careful reading and positive feedback on our paper. We have addressed the reviewer concerns and questions below, and we believe our proposed revisions significantly strengthen the manuscript.

We would be grateful if the reviewer might consider increasing their score in case they are satisfied with the proposed changes.

• W1. Lack of real-world datasets. We agree with the reviewer that incorporating more complex datasets would strengthen our experimental validation. To address this, we have added empirical results on the dSprites dataset. This is a well-established, high-dimensional dataset for NPIV, serving as a crucial example of a complex scenario where our method demonstrates strong applicability and scalability. We refer to our answer to Weakness 1 of Reviewer m9NX for details.

• W2. Assumption of familiarity with advanced spectral operator theory. We agree that parts of the paper might assume a high degree of familiarity with advanced spectral operator theory, potentially limiting accessibility for a broader NeurIPS audience. To address this, we will add a dedicated section in the appendix to provide a concise and accessible introduction to the necessary concepts from spectral operator theory. This section will clarify key definitions and notations essential for understanding our theoretical framework, aiming to make the paper more self-contained and interpretable without burdening the main text.

• W3. Practical applicability and unobservable spectral properties. We agree that the unobservability of these properties in practice is a significant challenge. Crucially, since submission, we have developed a strategy to empirically quantify spectral alignment and eigendecay and have tested it on the dSprites dataset. We also show how to empirically identify the effective cutoff dimension ($d$). This provides a means to gauge the decay of singular values and thus the effective ill-posedness, helping to operationalize aspects of the "good," "bad," and "ugly" scenarios in practice. We refer to our answer to Question 2 & Weakness 1 of Reviewer m9NX for details.

  We will incorporate details of these empirical strategies into the paper, demonstrating how practitioners can gain insight into the spectral properties of their data, even without ground truth.

• Question. Absence of comparisons with established baseline methods. It is important to clarify that our paper primarily focuses on the theoretical characterization of spectral features and their performance regimes (good, bad, ugly) for NPIV. The spectral features framework, which our paper analyzes theoretically, has already been empirically benchmarked extensively against a range of other state-of-the-art methods by [1] (including KIV, DFIV, DeepGMM and Dual Embedding). That study provides comprehensive empirical evidence of its practical performance against established benchmarks. Therefore, a full-scale empirical comparison was not the main objective of the present theoretical contribution. We will add a sentence to the experimental section of the main paper to explicitly direct readers to this relevant work and its comprehensive benchmarks.

[1] Haotian Sun, Antoine Moulin, Tongzheng Ren, Arthur Gretton, Bo Dai "Spectral Representation for Causal Estimation with Hidden Confounders", AISTATS, 2025.

We thank the reviewer for their positive assessment of our contribution and valuable comments! We have addressed all points of concern and have proposed edits to the manuscript. We elaborate on the details below.

We would be grateful if the reviewer might consider increasing their score in the event that they are satisfied with the proposed changes.

• Q1. Robustness to violations of Assumption 1. As noted, estimating the structural function ($h_0$) in the NPIV model is equivalent to solving the ill-posed inverse problem $\mathcal{T}(h) = \mathbb{E} [Y∣Z]$ in $h$, where identifiability of $h_0$ hinges on the injectivity of the operator $\mathcal{T}$ (Darolles et al [9]). When $\mathcal{T}$ is not strictly injective, we face partial identifiability. In such scenarios, while unique identification of $h_0$ is lost, it is possible to establish consistency to a minimum-norm solution of the inverse problem. This approach is common in partially identified instrumental variable models (Bennett et al [2], Meunier et al [27]]).

  Regarding the second part of Assumption 1 (existence of a solution, $\mathcal{T}^{−1}(\{\mathbb{E}[Y∣Z]\}) \ne \emptyset$): This is a fundamental model specification assumption. A violation here implies that our NPIV model structure is misspecified for the data, meaning no true $h_0$ exists that satisfies the moment condition of the model. This is a much more severe form of "violation" than a lack of injectivity, as it implies the problem itself has no solution consistent with the model, and therefore cannot be "slightly" violated in a meaningful way for consistent estimation. Any prior work on NPIV relies on this assumption for the existence of a suitable $h_0$.

  Finally, we thank the reviewer for pointing us to anchor regression. It is crucial to clarify that anchor regression addresses a different challenge than violations of our Assumption 1. While our assumption pertains to the existence and injectivity of the operator $\mathcal{T}$ for the identifiability and consistent estimation of the structural function $h_0$ in the NPIV model, anchor regression is designed for scenarios where traditional instrumental variable assumptions, specifically the exclusion restriction, are violated (i.e., the instrument directly causes the outcome). This violation does lead to a loss of identifiability for the causal effect in classical IV setups. However, anchor regression's primary objective is distributionally robust prediction, providing a stable predictive model even when an 'instrument' is technically invalid, rather than focusing on the consistent identification of the underlying structural function as our NPIV framework does.

  For conciseness, we did not elaborate on this in the main text. However, we agree this is an important nuance and will add a remark after Assumption 1 to clarify this point and its implications for partial identifiability.

• Q2. Empirical choice of cutoff dimension (d). We agree that the optimal choice of the latent dimension $d$ (cutoff dimension) inherently depends on the decay of the operator's singular values, which is a priori unknown. This is a common challenge in ill-posed inverse problems and spectral methods, analogous to selecting the effective rank in noisy matrix completion. Theoretically, this can be addressed using concepts like the Baik–Ben Arous–Péché (BBP) transition [R1], which provide a formal prediction for the "edge" of the spectrum that arises purely from statistical noise. Any singular value found in the data that is significantly larger than this theoretical edge would be considered part of the signal. A simple and practical way to find this same signal-noise boundary is to perform a stability check: Repeatedly estimate the singular values on different bootstrap samples of the data. The estimates of the true "signal" singular values will be stable (low variance), while the "noise" values will be unstable (high variance). Then, choose $d$ at the "elbow" where this transition occurs. We refer to our answer to W1 below for further discussion.

• W1. Experimental evaluation, spectral scenarios. Empirically evaluating proxies for spectral alignment is indeed essential for assessing whether practical examples fall into the "good" spectral scenario.

  Quantifying spectral alignment in dSprites

  After submission, inspired by the numerical methods for linear inverse problems [R2], we developped a strategy to quantify the spectral alignment, $a_i = \langle h_0 ,u_i\rangle_{L^2(X)}$ where $u_i$ are the left eigenfunctions of $\mathcal{T}$: $\mathcal{T} = \sum_{i \geq 1} \sigma_i v_i \otimes u_i$. Coeficient $a_i$ directly measures the contribution of the $i$-th singular function to $h_0$ in the $L_2(X)$-norm, that is $\lVert h_0 \rVert_{L_2(X)} = \sum_i a_i^2$. Since $h_0$ is unknown, we exploit the identity:

  $$a_i = \frac{1}{\sigma_i} \langle h_0, \mathcal{T}^*v_i \rangle_{L_2(X)} = \frac{1}{\sigma_i} \langle \mathcal{T} h_0, v_i \rangle_{L_2(Z)} = \frac{1}{\sigma_i} \mathbb{E}[(\mathcal{T}h_0)(Z)v_i(Z)] = \frac{1}{\sigma_i} \mathbb{E}[\mathbb{E}[Y \mid Z]v_i(Z)] = \frac{1}{\sigma_i} \mathbb{E}[Yv_i(Z)]$$

  where we used the singular function relationship $\sigma_i u_i = \mathcal{T}^*v_i$ and the fact that $h_0$ satisfies $\mathcal{T}h_0=\mathbb{E}[Y \mid Z]$.

  This identity shows we can estimate the coefficient $a_i$ by estimating the singular value $\sigma_i$ and the left singular function $v_i$ from $(Z, X)$ samples, and then estimate $a_i$ using $(Y, Z)$ samples. More specifically, the procedure is as follows.

  1. Learn spectral features using the spectral contrastive method from our paper.
  2. Estimate SVD components: Perform a canonical correlation analysis (CCA) on these features to get estimates of the leading singular values $\hat \sigma_i$ and singular function $\hat v_i$.
  3. Quantify stability: Perform subsampling and get a measure of uncertainty for our estimates (e.g., define confidence sets using quantiles). One would typically expect the estimates for the leading components to be stable, while uncertainty grows for components corresponding to smaller singular values.
  4. Estimate alignment: using these stable estimates, calculate the proportion of the $L_2$ norm of $h_0$ captured by the leading features.

  Applying this procedure to the dSprites dataset reveals:

  • The singular value spectrum is initially flat and then decays sharply, indicating a spectral structure that exhibits the BBP-type transition discussed in our response to Q2.
  • Our alignment estimates show that approximately 85% of the squared norm of $h_0$ is explained by the top 10 features, and about 95% by the top 20.

  Together, these findings provide evidence that the dSprites experiment falls into the "good" regime. This methodology provides a practical tool for assessing spectral properties in real-world settings. We will add a full description of this procedure and the new results to the appendix.

  Demand Design dataset

  The Demand Design dataset includes observed confounding variables. This leads to the conditional mean operator being non-compact. Consequently, direct notions of spectral alignment and decay for $\mathcal{T}$ become significantly more involved and are outside the scope of our current work, though an interesting and important problem for future research.

  Illustrating bad and ugly scenarios

  Illustrating the "bad" and "ugly" scenarios on real datasets is an important direction. Our primary goal in this work was to first establish a clear theoretical framework for these failure modes and validate their existence in a controlled setting. The empirical tools we have now developped (as discussed for dSprites) make it feasible to search for these scenarios in practice.

  While we have not yet run specific experiments, we can describe plausible bad/ugly real-world situations:

  • "Bad" scenario (weak instrument): this is a well-known challenge in applied econometrics, the bad regime can arise when an instrument has only a weak effect on the treatment. For instance, using small, local policy variations as instruments for nationwide economic behavior often leads to this issue.
  • "Ugly" scenario: this failure mode can occur if the instrument influences the treatment through a mechanism that is irrelevant to the structural function. For example, if an instrument (like an insurance plan) predicts treatment (a drug), but the health outcome depends on a factor (like a gene) that is uncorrelated with the instrument, then the estimation of the structural function would fail because the instrument provides the "wrong" kind of variation in the treatment.

• W2. Lower bound. We agree with the point made on the lower bound. As explained in our answer to Q1 of Reviewer 9cMm, we can now show that our upper bound in Corollary 1 is tight, addressing the optimality of spectral methods and the fundamental nature of our proposed scenarios. This will be added as a remark, with full details in the appendix.

• W3. Control on $\varepsilon$. We agree that the absence of precise characterization of $\varepsilon$ in Proposition 3 weakens interpretability. However, representation learning is challenging enough on its own, and uncovering tight theoretical bounds for $\varepsilon$ would require a dedicated and substantial investigation (e.g., a full paper on spectral contrastive learning generalization), which lies beyond the scope of our current work. We will add a remark near Proposition 3 to explicitly highlight this as an important and open problem for future research.

[R1] Baik et al. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Annals of Probability, 2005

[R2] Hansen et al. Deblurring Images: Matrices, Spectra, and Filtering, SIAM, 2006

We thank the reviewer for the time spent in reviewing our manuscript, for the encouraging evaluation of our work, and for the helpful feedback. We address the questions and concerns of the reviewer below, and have proposed updates to the manuscript to address the latter.

We would be grateful if the reviewer might consider increasing their score in case they are satisfied with the proposed changes.

• W1. Regarding the theoretical contribution. While the high-level intuition that spectral alignment and singular value decay matter seems natural in retrospect, this understanding remained at the level of folklore, lacking a rigorous treatment. Our primary contribution is to systematize this intuition, characterizing the different performance regimes and, crucially, the failure modes of spectral feature learning for NPIV. By establishing a concrete "good-bad-ugly" taxonomy, along with a method to empirically evaluate which case a given problem falls into (see our answer to Weakness 1 of Reviewer m9NX), we offer valuable insights to both theorists and practitioners.

• W2. Regarding the use of real-world data. We used synthetic data to cleanly isolate the effects of spectral alignment and singular value decay, as these ground-truth properties are unknown in real-world settings. We agree that validating our taxonomy on more complex datasets is an important step. As detailed in our response to Weakness 1 of Reviewer m9NX, we have a procedure to estimate spectral properties from data and have applied it to the dSprites dataset, confirming it falls into the "good" regime. We will add these results to the final manuscript.

• Q1. Comparison between misalignment and fast decay. We split our answer in two parts: first, assuming we work with the true singular functions of $\mathcal{T}$ as features; second, when we work with learned features obtained from our constrastive learning procedure.

  Assuming one has access to the true normalised features (i.e. the singular functions) $u_i,v_i$ of the conditional mean operator $\mathcal{T} =\sum_{i \geq 1 }\sigma_i v_i \otimes u_i$. The overall generalization error rate for $h_0$ depends on two key factors:

  • Operator ill-posedness: The rate of singular value decay of $\mathcal{T}$, typically $\sigma_i \asymp i^{-p}$ for $p \geq 1$. A larger $p$ indicates faster decay and thus greater ill-posedness.

  • Solution smoothness/alignment: The source condition

  $$\left\lVert h_0 \right\rVert^2_{r} := \left\lVert (\mathcal{T}^*\mathcal{T})^{-r/2}h_0 \right\rVert^2 = \sum_{i\geq 1} \frac{\langle h_0, u_i \rangle_{L_2(X)}^2}{\sigma_i^{2r}} < +\infty$$

  This condition, for $r \geq 0$, quantifies how "smooth" $h_0$ is relative to the spectral decomposition of $\mathcal{T}$, or equivalently, how quickly the alignment coefficients $\langle h_0, u_i \rangle_{L_2(X)}$ decay relative to the singular values. A larger $r$ implies $h_0$ is "smoother" or better aligned with the less ill-posed directions of $\mathcal{T}$. Note that under the "bad" scenario where the eigendecay is fast ($p$ is large), a stronger alignment is necessary for $\left\lVert h_0 \right\rVert^2_{r}$ to be finite for a large value of $r$. While in the "good" scenario with $p \sim 1$, $\left\lVert h_0 \right\rVert^2_{r} < +\infty$ is more likely to hold with a large value of $r$.

  Under the source condition, the approximation error when using as features the top-$d$ singular functions of $\mathcal{T}$ is:

  $$\sum_{i>d} \langle h_0, u_i \rangle_{L_2(X)}^2 \leq \sigma_d^{2r} \sum_{i>d} \frac{\langle h_0, u_i \rangle_{L_2(X)}^2}{\sigma_i^{2r}} \leq \sigma_d^{2r}\left\lVert h_0\right\rVert^2_{r}.$$

  Combined with the eigenvalue decay (involving $p$), we obtain from Corollary 1 in our submission that:

  $$\left\lVert \hat{h}-h_0\right\rVert_{L^2}=O_p\left(\left\lVert h_0 \right\rVert_{r}\sigma_d^{r}+\sqrt{\frac{d}{n\sigma_d^2}}\right) = O_p\left(\frac{\left\lVert h_0 \right\rVert_{r}}{d^{r\times p}}+d^{1/2 +p}\sqrt{\frac{1}{n}}\right).$$

  Balancing both terms in $d$ as a function of $n$, we obtain

  $$\left\lVert\hat{h}-h_0 \right\rVert_{L^2}=O_p\left(n^{-\frac{r}{2(r+1) + p^{-1}}}\right).$$

  with $d(n) = n^{\frac{1}{1 + 2p(r+1)}}$. This bound is tight as a matching lower bound is obtained in [1, Theorem 1-(i)].

  From this tight rate, we see that both the ill-posedness ($p$) and the relative smoothness ($r$) fundamentally damage the effectiveness of 2SLS, and their impacts are inherently intertwined.

  • A fast decay (large $p$) directly slows down the rate regardless of $r$. It means singular values drop quickly, making it fundamentally harder to resolve higher-frequency components of $h_0$.
  • A low relative smoothness (small $r$) means $h_0$ has significant components aligned with directions corresponding to very small singular values, even after accounting for the singular value decay. This also directly slows the rate, effectively rendering the problem very hard or even impossible ($r \to 0_+$, makes the rate vacuous, representing our "ugly" scenario).

  When working with learned features via contrastive learning. We see from our Proposition 3 that when we learn the spectral features using contrastive learning, we incur an additional penalty $\varepsilon/\sigma_d$ in the upper bound, where $\varepsilon$ is the optimality gap on the contrastive learning loss. The alignment of $h_0$ does not enter this factor, as learning the eigensubspaces of $\mathcal{T}$ is unrelated to $h_0$. This term shows that a faster eigendecay makes the weight of the error incurred by contrastive learning grow faster. Intuitively, features corresponding to small singular values can be empirically difficult to learn, even with a very large number of $(Z,X)$ samples.

  We will add this discussion as a remark in the main paper, with full details in the Appendix. We think that it is a great addition to the discussion in our manuscript.

[1] Xiaohong Chen, Markus Reiss. "On rate optimality for ill-posed inverse problems in econometrics." Econometric Theory, Vol. 27, No. 3, INVERSE PROBLEMS IN ECONOMETRICS (June 2011), pp. 497-521.