Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regression

Thank you for reviewing our paper and providing valuable suggestions! Below are our responses. We would be happy to further discuss any additional questions that might arise. If we have satisfactorily addressed your questions, we would be grateful if you would consider increasing your score.

Response to Weaknesses.

W1. Compared to existing analyses for ordinary regression, the two-stage regression problem presents the following new technical challenges. We will make this clearer in the main text.

• We need to prove a novel general two-stage oracle bound (44), which is key to all DFIV upper bounds. The proof is highly nontrivial and involved, and is detailed throughout Appendix D.1-D.6. A proof sketch is also given in Appendix B. The resulting bound is natural and can be applied to any two-stage risk minimization estimator.
• Since Stage 1 is effectively learning the $T$-projected Stage 2 network, the regularity of the DNN estimator must be controlled to analyze the Stage 1 error, which is a new consideration in this line of analysis. This leads to the discussion of smoothness in Section 3.1.
• A new smooth approximation analysis for sigmoid DNNs involving the Besov norm is needed, which we develop in Appendix C.
• Our linear IV minimax lower bounds (Theorems 4.1 and F.2) are also new and establish a novel separation. In comparison, our minimax lower bounds (Proposition 3.3 and 3.7) generalize well-known results in the IV literature (for Sobolev balls, RKHS, etc) to Besov spaces.

W2. In the IV literature, DFIV has already been empirically shown to perform well at a variety of tasks. In [1], Section 4.3 and [2], Section 5.2, DFIV is shown to outperform competitors including Kernel IV on demand design and policy evaluation tasks in reinforcement learning such as Behaviour Suite tasks. For example, Mountain Car requires the agent to accelerate right on the left side of the valley and left on the right side to gain momentum, which results in a discontinuous target function. This closely aligns with our results on Section 4, which predicts that DFIV outperforms kernel estimators for target functions with both high and low smoothness regions.

W3. Our work focuses on sample complexity rather than optimization analysis. There is no satisfactory optimization theory for DNNs even in the basic supervised setting, so understanding the iterative optimization of two separate DNNs is extremely challenging. Nevertheless, understanding the statistical properties of the ideal solution is still an important problem as optimization can only guarantee that the gap from the ideal solution converges to zero as training progresses.

Practical issues concerning convergence of DFIV have been discussed extensively in [1]. For example, in Appendix F they also considered joint optimization of the Stage 2 loss but found this to be too unstable. Alternatively, a recent paper [3] develops a functional bilevel optimization algorithm with convergence guarantees which can be directly applied to IV regression (Section 4.1). We are also developing a follow-up work analyzing the dynamics of joint optimization for two-stage regression in e.g. two-layer NN settings.

Response to Questions.

We thank the reviewer for the interesting technical question. Indeed, our smoothness assumption is closely related to the $\beta$-source condition popular in the nonparametric statistics literature and also the NPIV literature [4][5]. To explicitly explain the relationship, we provide an example in the kernel regression scenario.

Suppose our target function lives in a Besov space with smoothness parameters $s,2,2$ i.e. $f_{str} \in B_{2,2}^s(X)$ and we employ a Matern kernel $k_r$ with parameter $r \geq d_x/2$, inducing a reproducing kernel Hilbert space (RKHS) $H_r(X)$. Furthermore, suppose that the marginal distribution is equivalent to the Lebesgue measure, i.e., the density $p$ is both upper and lower bounded by some constant. Define the integral operator $L$ acting on functions as $Lf(x) = \int k_r(y,x)f(y) dp(y)$. The $\beta$-source condition used in nonparametric kernel learning reads $\|L^{-\frac{\beta}{2}}f\|_{L_2(p)} \leq B$.

One can then show that $f_{str} \in B_{2,2}^s(X)$ is equivalent to $\|L^{-\frac{\beta}{2}}f_{str}\|_{L_2(p)} \leq B$ with $\beta = \frac{s}{r}$ [6].

[1] Xu et al. Learning Deep Features in Instrumental Variable Regression. ICLR 2021.

[2] Chen et al. On Instrumental Variable Regression for Deep Offline Policy Evaluation. JMLR 2022.

[3] Petrulionyte et al. Functional Bilevel Optimization for Machine Learning. NeurIPS 2024.

[4] Chen et al., On Rate Optimality For Ill-posed Inverse Problems in Econometrics. Econometric Theory 2011.

[5] Chen and Christensen. Optimal Sup-norm Rates and Uniform Inference on Nonlinear Functionals of Nonparametric IV Regression. Quantitative Economics 2018.

[6] Fisher and Steinwart. Sobolev Norm Learning Rates for Regularized Least-Squares Algorithms. JMLR 2020.

Thank you for your detailed review and insightful suggestions, which have helped us greatly to improve the manuscript! Below are our responses. We would be happy to further discuss any additional questions that might arise. If we have satisfactorily addressed your questions, we would be grateful if you would consider increasing your score.

Response to Weaknesses.

W1. We have added a Table of Symbols at the beginning of the appendix. Also, a diagram illustrating the causal graph of Eq.(2) has been added to the beginning of the introduction.

W2.

• Eq.(2) expresses the basic setting of IV regression, which is always assumed in modern formulations of IV. Please see Figure 1 for an illustration.

• For Equation (3) to (4), and the role of the mapping $T$, please see our answer to Question 6.

• For Definition 2.2: The B-spline basis is a natural hierarchical basis for Besov spaces. It has been extensively studied in the context of approximation theory and multiresolution analysis due to its localized nature (as opposed to e.g. Fourier basis which is used for global frequency analysis), see e.g. [1]. More recently, it has also been used to analyze the sample complexity of ordinary regression with DNNs [2]. We require a concrete basis to conduct smooth approximation analysis (Appendix C) and compute minimax rates. The localized nature of B-splines also helps us derive the separation with kernel methods (Section 4 and Appendix F.4).

Response to Questions.

Q1. The additional assumption refers to Assumption 4 on the smoothness of $T$. We have mentioned its necessity in the abstract (it essentially allows us to control the difficulty of the Stage 1 regression).

Q2. A diagram illustrating the causal graph of Eq.(2) has been added to the beginning of the introduction, see Figure 1.

Q3. Indeed, adaptivity and efficiency refer to two different key aspects of our contributions. Adaptivity (of features w.r.t. data) fundamentally enables DFIV to achieve the optimal rate when the target function possesses low spatial homogeneity, while any fixed-feature method is provably worse; these results are presented in Section 4. Efficiency in the Stage 1 samples means that $m\approx n$ suffices to achieve the optimal rate in $n$, while previous studies on kernel IV all require $m\gg n$; this is proved in Corollary 3.4, 3.8 and explained in depth in the Optimal splitting paragraph (line 450-455). Both results are novel improvements over existing NPIV studies to the best of our knowledge.

Q4. We have added that the functional equation follows from taking the conditional expectation w.r.t. $Z$. (Referencing Eq.(4) is also possible, but we feel it is better to explain using words since Eq.(4) comes later in the text.)

(continued)

(continued)

Q5. Compared to existing analyses for ordinary regression, the two-stage regression problem presents the following new technical challenges.

• We need to prove a novel general two-stage oracle bound (44), which is key to all DFIV upper bounds. The proof is highly nontrivial and involved, and is detailed throughout Appendix D.1-D.6. A proof sketch is also given in Appendix B. The resulting bound is natural and can be applied to any two-stage risk minimization estimator.
• Since Stage 1 is effectively learning the $T$-projected Stage 2 network, the regularity of the DNN estimator must be controlled to analyze the Stage 1 error, which is a new consideration in this line of analysis. This leads to the discussion of smoothness in Section 3.1.
• A new smooth approximation analysis for sigmoid DNNs involving the Besov norm is needed, which we develop in Appendix C.
• Our linear IV minimax lower bounds (Theorems 4.1 and F.2) are also new and establish a novel separation. In comparison, our minimax lower bounds (Proposition 3.3 and 3.7) generalize well-known results in the IV literature (for Sobolev balls, RKHS, etc) to Besov spaces.

Q6. The original problem Eq.(3) cannot be solved directly by e.g. minimizing $\lVert Y-\hat{f}(X)\rVert^2$ since $X$ is correlated with the noise; it will result in a biased estimator, i.e. $\hat{f}$ will not converge to $f$ even with many samples. Reformulating this as Eq.(4) by isolating the $Z$-independent noise component shows that the data can be essentially seen as generated from the projected function $Tf(Z)$. (Note that this is an equivalent reformulation, so we are not making the problem harder to solve in any way.) This motivates the family of two-stage estimators: learn $T\hat{f}$ in Stage 1 and use this to minimize $\lVert Y-T\hat{f}(Z)\rVert^2$ in Stage 2. This does result in an asymptotically unbiased estimator and in fact achieves the optimal convergence rate as we show in our paper.

Moreover, this shows that $T$ also affects the difficulty of learning (as we essentially receive data from the projection $Tf$ rather than $f$), which is why the optimal rates depend on the spectral decay rate of $T$. As such, the transformation Eq.(4) and the operator $T$ are fundamental to the IV analysis literature; please see [3][4][5] and the references within for further details.

[1] Ronald A. DeVore and Vasil A. Popov. Interpolation of Besov spaces. Transactions of the American Mathematical Society, 305(1):397–414, 1988.

[2] Taiji Suzuki. Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality. ICLR 2019.

[3] Xiaohong Chen and Markus Reiss. On rate optimality for ill-posed inverse problems in econometrics. Econometric Theory, 27(3):497–521, 2011.

[4] Whitney K. Newey and James L. Powell. Instrumental variable estimation of nonparametric models. Econometrica, 71(5):1565–1578, 2003.

[5] Serge Darolles, Yanqin Fan, Jean-Pierre Florens, and Eric Renault. Nonparametric instrumental regression. Econometrica, 79(5):1541–1565, 2011.

Most of considerations have been resolved, and I will increase my score.

Thank you for your positive assessment of our contribution and valuable comments! Below are our responses.

Q1. While unrolling is conceptually possible, the original DFIV paper [1] states "...we do not attempt to backpropagate through the estimate $\hat{\theta}_z$ to do this, however, as this would be too computationally expensive." They also considered joint optimization of the Stage 2 loss w.r.t. $\hat{\theta}_x, \hat{\theta}_z$ but found this to be too unstable. Alternatively, a recent paper [2] develops a functional bilevel optimization algorithm with convergence guarantees which can be directly applied to IV regression (Section 4.1).

Q2. Integrability is not an issue since we assumed the support of $X,Z$ is compact at the beginning, although this can be relaxed to the noncompact case by assuming suitable tail decay; hence any lower bounded density suffices. Moreover, Assumption 5 is only required for the projected minimax lower bound (Prop 3.3). It is not needed for the full minimax lower bound (Prop 3.7) nor the DFIV upper bounds (Thms 3.1, 3.5) which are our key contributions.

Intuitively, minimax lower bounds are proved by constructing large well-separated subsets of the target space. Assumption 5 is needed in Prop 3.3 to translate the separation in the projected MSE, only guaranteed in $L^2(P_X)$-norm due to how the reverse link condition is defined, to $L^2(X)$-norm in line 2012. It is not needed for the full lower bound since we are not interested in the projected MSE. (To be completely precise, since the full upper bound is given in the $L^2(P_X)$-norm and the full lower bound is given in the $L^2(X)$-norm, Assumption 5 is also needed to conclude that the rates match, but this is just a technicality of minimax analysis.)

Q3. The minimum of Stage 1 always exists since the DNN parameter space is compact (all parameters in $\theta_z$ are bounded above by $M$) and the loss depends continuously on the DNN output. The minimum of Stage 2 is not guaranteed to exist as the output $\hat{\theta}_z$ of Stage 1 may not depend continuously on $\theta_x$, but in this case we may simply take an approximate minimizer $\epsilon$-close to the infimum of the loss and obtain the same result by taking $\epsilon$ small enough. This remark has been added to the main text.

The underlying assumption that the empirical risk minimizer can be found is unavoidable in our sample complexity analysis. There is no satisfactory optimization theory for DNNs even in the basic supervised setting, so understanding the iterative optimization of two separate DNNs is extremely challenging. Nevertheless, understanding the statistical properties of the ideal solution is still an important problem as optimization can only guarantee that the gap from the ideal solution converges to zero as training progresses. We are also developing a follow-up work analyzing the dynamics of joint optimization for two-stage regression in e.g. two-layer NN settings.

Q4. While the smoothness restriction/regularization is admittedly unusual, there are similar precedents in neural network training; please see the references on line 348-350. Moreover, the norm is easy to estimate if $s$ is an integer since it is equivalent to the Sobolev norm. This can be obtained by automatically computing the order $s$ derivatives of the network w.r.t. the input and numerically integrating. If $s$ is not an integer, the norm can still be computed with the discretization (line 339) (which can be backpropagated through) or bounded above by the order $\lceil s\rceil$ Sobolev norm (which suffices for the restriction-based approach).

Q5. In the IV literature, DFIV has already been empirically shown to excel at a variety of tasks, in particular RL/control tasks with discontinuous/low smoothness target functions ([1], Section 4.3 and [3], Section 5.2), aligning with our results. While additional experiments on the smoothness aspect could be interesting, we have elected to focus on the perspective of statistical learning theory. We also mention that sample complexity papers, such as the original paper on DNN regression [4], usually do not conduct new experiments.

Minor: Thank you for pointing out the mistakes, they have been fixed. Line 368 indeed means that $\log m,\log (m\wedge n)$ must be replaced by $(\log m)^3, (\log (m\wedge n))^3$, respectively.

[1] Xu et al. Learning Deep Features in Instrumental Variable Regression. ICLR 2021.

[2] Petrulionyte et al. Functional Bilevel Optimization for Machine Learning. NeurIPS 2024.

[3] Chen et al. On Instrumental Variable Regression for Deep Offline Policy Evaluation. JMLR 2022.

[4] Taiji Suzuki. Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality. ICLR 2019.