We thank the reviewer for raising concerns regarding the specificity of the assumptions underlying our method. We agree that the lazy regime and Neural Tangent Kernel (NTK) theory represent specific training dynamics, and we would like to highlight that part of our contributions is the development of practical tools to validate these assumptions, allowing practitioners to determine precisely the scenarios in which Algorithm 1 operates effectively.

A significant challenge in utilizing methods grounded in the lazy regime and low-rank assumptions is understanding when these properties hold. To address this, we propose empirical methods to:

1. Quantify the laziness of a neural network by measuring the relative change in the Jacobian using the ratio $\frac{\|\nabla_w^2 f\|}{\|\nabla_w f\|^2}$, as described in Section 3.2. This enables practitioners to determine if a given network operates in the lazy regime.

2. Verify the low-rank structure of the Jacobian by estimating its effective rank using a reference set of size $r$, leveraging the reduced-dimensional covariance matrix $J_R^T J_R$. This ensures scalability even in large models, as shown in our experiments in Section 4.1.

These validation techniques are integral to our method, enabling users to assess the applicability of Algorithm 1 to their specific architectures and datasets. For example:

• For tasks where the lazy regime and low-rank structure hold, such as MNIST (Figure 3), our method achieves notable performance while significantly reducing computational costs.
• If the assumptions do not hold (e.g., tasks requiring extensive feature adaptation), the validation tools provide early indicators, ensuring that Algorithm 1 is applied only in suitable scenarios.

Thus, while the lazy regime and NTK theory may not universally apply, our contribution extends beyond the algorithm itself to include a framework for verifying these assumptions in practice. This framework not only supports the effective application of Algorithm 1 but also advances the understanding of when and where these assumptions are valid.

We hope this explanation clarifies that our work explicitly addresses the concern of assumption validity by providing tools to empirically determine the scenarios where the method operates effectively. This contribution enhances the practical utility of our approach while expanding its theoretical foundations.

We appreciate the reviewer’s observation regarding the generalization properties of the lazy regime and the potential concerns raised in [1]. While it is true that the lazy regime has been associated with suboptimal generalization in some settings, it is important to contextualize its applicability within the broader landscape of neural network training.

First, the lazy regime is not inherently something to "avoid," as its utility depends heavily on the specific application and task. For example, in large language models (LLMs) and other overparameterized architectures, the lazy regime has been leveraged successfully to simplify training dynamics. The constancy of the Neural Tangent Kernel (NTK) in this regime provides provable convergence guarantees, which are critical in large-scale settings where computational efficiency and optimization stability are priorities.

Second, while generalization in the lazy regime may be suboptimal for complex tasks requiring feature learning, recent works, such as Geiger, Mario et al. (2020) [11], demonstrate that sufficiently wide networks operating in the lazy regime can still achieve strong performance on simpler tasks, including MNIST and CIFAR. These findings indicate that the lazy regime, despite its limitations, can be effectively utilized in scenarios where the task complexity aligns with its capabilities.

In our work, we do not advocate for the lazy regime as a universal solution but rather leverage its properties to address specific challenges in training overparameterized models. By combining the lazy regime with the low-rank structure of the Jacobian, our approach achieves significant computational efficiency without compromising performance on tasks that align with this regime’s strengths. Empirical results in Section 4 confirm this, showing achieved accuracy while reducing training costs.

Thus, the lazy regime is not a phenomenon to categorically "avoid" but a tool that, when applied judiciously, can offer unique advantages in terms of computational efficiency and training stability for certain tasks and architectures.

We thank the reviewer for their comments regarding the experimental evidence and the role of the Jacobian in our training algorithm. To address the specific concern about the update of the Jacobian in Algorithm 1, we would like to clarify the theoretical foundation that underpins our approach.

The constancy of the Jacobian is a core assumption of the lazy regime, as established in Neural Tangent Kernel (NTK) theory (Section 2). In this regime, the network operates near its initialization, with minimal parameter updates during training. Consequently, the Jacobian remains approximately constant throughout the optimization process, as supported by the first-order Taylor expansion of the model (Eqn. 4). This eliminates the need for recalculating or updating the Jacobian during training, which is a cornerstone of our backpropagation-free algorithm.

In Algorithm 1, the Jacobian is precomputed during the preprocessing phase and reused throughout training without requiring updates. This design is both theoretically justified and computationally advantageous. The NTK framework ensures that, for sufficiently wide networks, the constancy of the Jacobian provides provable convergence guarantees, allowing us to replace dynamic updates with a single, efficient computation at initialization.

Empirical results in Section 4 further validate the practical feasibility of this assumption. For example, we demonstrate performance on MNIST under the lazy regime, where the precomputed Jacobian suffices to achieve notable accuracy. These results highlight that, under the appropriate settings, the near-constant Jacobian assumption not only simplifies training but also maintains effectiveness.

We hope this explanation clarifies that the lack of Jacobian updates in Algorithm 1 is not an oversight but a deliberate design choice grounded in NTK theory and validated by experimental results.

We appreciate the reviewer’s concern regarding the computational cost of eigenvector computation in Algorithm 1 and its efficiency on high-dimensional tasks. We would like to clarify that the algorithm is explicitly designed to mitigate such costs by leveraging the low-rank structure of the Jacobian, ensuring scalability even in large networks.

The key point to emphasize is that Algorithm 1 does not perform principal component analysis (PCA) or eigendecomposition on the full Jacobian matrix of size $m \times n$, where $m$ is the number of parameters and $n$ is the number of data points. Instead, we utilize a reference set $R$ of size $r \ll n$, allowing us to reduce the computational burden by performing eigendecomposition on the much smaller $r \times r$ matrix $J_{R}^T J_{R}$, as described in Proposition 3.1 and Section 3.3 of the paper.

To elaborate:

1. Low-Rank Assumption: The low-rank assumption ensures that the essential variations of the Jacobian are captured by its leading eigenvectors, which are derived from $J_{R}^T J_{R}$. This matrix is of size $r \times r$, where $r$ is the size of the reference set, and not $m \times n$, making the eigenvector computation both memory- and computationally-efficient.

2. Computational Complexity: The computational complexity of eigendecomposition is $O(r^3)$, which is orders of magnitude smaller than the cost of direct eigendecomposition of the full Jacobian, $O(\min(n, m)^3)$, in typical large-scale settings. For instance, even in networks with millions of parameters, selecting $r = 100$ leads to an efficient computation that is practical for high-dimensional tasks.

Additionally, the algorithm’s efficiency in high-dimensional tasks stems from its ability to decouple computation from input size. The use of a precomputed and reduced-dimensional Jacobian ensures that weight updates are performed efficiently without the need for iterative gradient computations over the full dataset. This advantage is particularly significant in tasks involving high-resolution images (e.g., CNNs) or long sequences (e.g., transformers), where input dimensionality traditionally imposes significant computational overhead in standard backpropagation.

Finally, empirical results (Figure 3) validate the scalability of the algorithm. These results demonstrate that even for networks with widths exceeding $1$ million parameters, the computational efficiency and accuracy of Algorithm 1 remain robust, showcasing its suitability for high-dimensional tasks.

We hope this clarification addresses the reviewer’s concerns and highlights the practical scalability of Algorithm 1 in handling high-dimensional neural networks.

We appreciate the reviewer’s perspective regarding the novelty of Algorithm 1 and welcome the opportunity to elaborate on its innovative contributions. Algorithm 1 is indeed novel, particularly in its ability to significantly reduce the computational burden associated with traditional backpropagation, leveraging both the lazy regime and the low-rank structure of the Jacobian.

First, we underline that while backpropagation involves layer-by-layer computation of gradients using the chain rule, Algorithm 1 bypasses this recursive mechanism by precomputing and reusing a reduced-dimensional representation of the Jacobian, denoted as $\mathbf{J}$. In parameter-sharing network architectures, such as CNNs, where the number of parameters $m$ and the input dimensions $n$ are immense, the computational cost of traditional backpropagation scales with $O(n \cdot m)$. In contrast, Algorithm 1 achieves a computational complexity of $O(m + n)$, as shown in Section 3.3 of our paper.

This efficiency stems from the following key innovations:

1. Low-Rank Approximation of the Jacobian: By utilizing a reference set $R$, we reduce the Jacobian’s dimensionality from $n \times m$ to $n \times r$, where $r \ll n$ (as detailed in Proposition 3.1). The eigenvectors of the reduced covariance matrix $\mathbf{J}^T \mathbf{J}$ retain the most significant information while discarding redundancy.

2. Precomputation and Reuse: Once the reduced Jacobian is precomputed in the initialization phase, weight updates during training are calculated using the approximation

   $$\mathbf{g}_{\text{approx}} = \mathbf{\tilde{J}} \mathbf{V}_J \mathbf{V}_J^T \mathbf{a}_t,$$

   where $\mathbf{V}_J$ are the principal components derived from the low-rank approximation. This avoids the need for repeated forward and backward passes through the network, significantly reducing computational overhead.

For example, in CNNs, where convolutional filters are applied across spatial dimensions, the complexity of backpropagation scales with the product of the feature map dimensions and the filter size. Algorithm 1 mitigates this by replacing layer-specific gradient computations with global, reusable projections derived from the reference set. Empirical results (Figure 3) validate these claims, showing that Algorithm 1 achieves comparable performance with reduced computational cost.

Finally, the theoretical grounding of Algorithm 1 is rooted in NTK theory, where minimal parameter updates in the lazy regime ensure that the precomputed Jacobian remains valid throughout training. By integrating the low-rank structure with lazy regime assumptions, Algorithm 1 uniquely combines these concepts to enable scalable, backpropagation-free training—a capability not demonstrated by prior works.

We believe these elements collectively demonstrate the algorithm’s novelty and practical relevance in addressing the computational challenges of training modern neural networks.