Glocal Hypergradient Estimation with Koopman Operator

Thank you for the insightful comments. Below, we answer your questions and concerns.

• W1: The existence of unit eigenvalues is derived from the assumption of the stable hypergradient after a long period of training. In the experiments, we tried to make the dynamics less noisy by using larger batch sizes.
• W2: We adopted simple image classification tasks to make the dynamics less stochastic.
• W3: The global method inevitably accumulates numerical errors over long-horizon, causing gradient explosion or vanishing (L149), similar to the training of pure RNNs. Since the proposed method relies on short-horizon information, it successfully avoids this issue. We will discuss this problem more carefully.
• W4: Thank you for suggesting another baseline that truncates the global method. We included the comparison in Fig C6 of the updated manuscript.

• Q1, Q2: Thank you for the comments. We updated the manuscript accordingly.
• Q3: We fixed the explanation as “if the dynamics involve no diverging modes” and the equation as $g(x_\infty)\approx \sum_{j: |\lambda_j|=1}\lambda_j^{t-1} \varphi_j(x_0) v_j.$
• Q4: $x_0, \dots, x_t$ is a trajectory of a dynamical system $x_{t+1}=f(x_t)$.
• Q5: In Eq 8, we discuss the state after convergence. For finite time steps $T$, the effect of modes with eigenvalues smaller than 1 is not negligible.
• Q6: This measurement function is applied to a batch of observations $[h_0, \dots, h_\tau]$ and converts it to $[k_0, \dots, k_{\tau-m+1}]$, where $k_t=[h_t,\dots,h_{t+m-1}]$. Thus, we do not need to compute $g(h_{t^*})$, where $t^*$ is the biggest index in $\mathcal{I}_s$. We updated the description accordingly.

Thank you for the helpful review. Below is our rebuttal to your questions and concerns.

W1: As the reviewer pointed out, this assumption does not always hold. However, it is generally used in many practical data-driven dynamical systems in various domains, and we followed it.

• W2: We believe the concept is simple: the core idea is to use the least square (Eq 9) to estimate a matrix that forwards the linearized dynamics of hypergradient. Practitioners can use tools to implement DMD, such as pyDMD.
• W3: Thank you for suggesting other experiments. We will add a new dataset. For models, we believe that LeNet and ResNet are sufficient to demonstrate the effectiveness of the proposed method on the datasets we used.
• W4: It would be appreciated if you elaborated more on what makes tables 2 and 3 unclear.

• Q1: We analyzed the complexities of DMD in Section 3.1 (around L298).
• Q2: If we augment the observation function as $(x, \mathbf{g}(x))$, then its left inverse is simply taking the first $\dim x$ elements, whose computational cost is negligibly small. Hankel DMD holds this property.
• Q3: The global method takes much longer computation than others, as explained in Table 3. Thus, we omit the results on non-MNIST datasets, as explained in L358.
• Q4: It is difficult to compare the gradient-based HPO methods with black-box ones directly. In terms of complexities, gradient-based ones are much more efficient because they can additionally use gradient information.

Thank you for the insightful comments. Below, we answer your questions and concerns.

• W1: How to set the dimension of an approximated Koopman operator $n$ and the dictionary $\mathbf{g}$ is problem dependent. In our experiments, we adopt Hankel DMD stacking 10 time-delayed vectors so that $n=10q$, where $q$ is the number of hyperparameters. 
• W2: Thank you for suggesting experiments. Since the proposed method is an online approach, it is hard to outperform or even match the performance of the best-tuned ones obtained through countless trials and errors by researchers for decades. In the updated manuscript, we added another baseline (Fig C6) to demonstrate that the proposed method shows better approximation than a truncated variant of the global baseline.
• W3: $n$ can be specified as written above.
• W4: Thank you for carefully reading our manuscript. $\phi$ is a typo of $\psi$, and we fixed it. We updated the description of Eq8. The connection between $\mathbf{g}$ and $g$ is explained around L186. 

• Q1: The training dynamics can be written as $\theta_{t+1}=\Omega(\theta_t)$, where $\theta_t$ is model parameters and $\Omega$ is the optimizer, so in theory it is possible to handle the dynamics. In practice, on the other hand, it would be challenging to accurately predict the dynamics after a significant loss drop only by using trajectory before the event without enough prior knowledge of the dynamics. However, the predicted states from dynamics before loss drop could help us to determine if the initial hyperparameters would lead to divergence of training dynamics or not.
• Q2: Figure C3 shows the results when the initial learning rate is 0.1. 
• Q3: Global HGD suffers from numerical instability, which appears after around 100 iterations.