MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters

We appreciate the reviewer's honesty in stating clearly that this paper falls outside their area of expertise. However, it seems there was a critical misunderstanding regarding the approximation in Equation (5).

Reviewer: "Using eqn (5) as a surrogate of eqn (4) is the basic to make the proposed method practical. However, such approximation is not reasonable. I cannot see any rationality in footnote 1 as justification..."

We clarify the approximation carefully below, as there appears to be a misunderstanding:

• The statement in Footnote 1 (p.2) specifically analyzes the case as $\eta \to 0$. In this scenario, indeed $\beta_T^5 - \beta_T^4 \to 0$. However, the non-trivial insight is that $(\beta_T^5 - \beta_T^4)/\eta \to 0$, meaning the difference between these terms vanishes faster than $\eta$. Concretely, this implies:

  • While both $\beta_T^4$ and $\beta_T^5$ scale roughly as $\eta T$, their difference scales strictly slower ($o(\eta T)$), making the approximation increasingly accurate for small $\eta$.

• Intuitively, each term in the RHS of Eq. (5) directly appears in Eq. (4), though at the earliest feasible time it can be computed (ensuring causality). Thus, Eq. (5) serves as a valid practical approximation to Eq. (4).

Finally, we emphasize that this forward-backward view approximation is well-established in related fields, notably in eligibility trace methods widely used in reinforcement learning literature.

We thank the reviewer for their thoughtful feedback and helpful suggestions, which we address below and plan to incorporate into the final version of the paper. We believe this will significantly enhance the clarity and quality of the work.

---

Continual Learning

I think the missing benchmark that would really complete this paper would be in the method's application to continual learning problems... the paper discusses that MetaOptimize can be applied to continual learning problems but... it is not evaluated on any.

We already include a continual learning experiment in Section 7.2 on continual CIFAR-100 benchmark. This benchmark sequentially trains on 10 tasks (10 classes each) without explicit task boundaries or weight resets. As discussed in the paragraph preceeding Figures 3 and 4, MetaOptimize demonstrates substantial advantages in continual learning, including fast adaptation to new tasks, and favourable layer-wise adaptive behaviour.

I would love to see more discussion around figure 3 and 4—in particular, why you think this happens.

Here are some intuition for the observed phenomena:

• Why later-layer step-sizes are consistently larger:
  This aligns with established best practices in supervised learning (Howard & Ruder, 2018), where larger step-sizes in later layers accelerate training. MetaOptimize autonomously discovers this pattern, despite equal initial step-sizes (10⁻⁴).

• Why step-sizes of earlier layers decrease and later-layers increase in continual learning:
  Early layers converge toward stable low-level image features shared across tasks, thus needing smaller updates over time. Later layers must continually adapt to changing labels, thus requiring constantly larger step-sizes.

• Saw-tooth pattern in step-sizes (Fig. 4):
  Each spike corresponds exactly to task transitions. MetaOptimize momentarily boosts step-sizes to facilitate rapid adaptation to new tasks.

I would want to see this phenomenon occur... over multiple seeds...

Figures 3 and 4 report averages over 5 random seeds, an important detail we mistakenly omitted in the submitted manuscript. We will explicitly clarify this in our revision.

---

Methods and Evaluations

The one comparison I think is missing is between their implementation and the specific instances of their framework (i.e. hypergradient descent and IDBD).

Thank you for pointing this out. Hypergradient Descent is already included in our experiments as gdtau (Chandra et al., 2022), an efficient implementation of hypergradient descent.

Regarding IDBD, its original design targets linear regression tasks, making it inapplicable to the neural-network-based scenarios presented in our work.

In figure 3, average cumulative accuracy is a very strange metric. Why not report the accuracy?

Average cumulative accuracy is used in continual learning mainly because it summarizes algorithmic performance across multiple tasks, avoiding difficult/misleading interpretations from task-specific accuracy variations. In the final version of the paper, we will also include the accuracy curves to reveal such variations.

figure 4 is a bit misleading, since the two lines use different scales.

Thank you for highlighting this. Initially, both blocks have identical step-sizes (10⁻⁴). To eliminate confusion, we will plot step-sizes for the two blocks in separate subfigures.

---

Relation to Broader Scientific Literature (Learned Optimization)

Thank you for highlighting the interesting relevant works on Learned-to-Optimize (L2O). Traditional L2O approaches typically learn optimizers or hyperparameters offline, then fix them at deployment. In contrast, MetaOptimize dynamically adjusts hyperparameters (step sizes) concurrently during training. This real-time adaptation to changing conditions or tasks is specifically beneficial in continual learning. In the revised manuscript, we will carefully contrast MetaOptimize with L2O, explicitly citing and discussing the references you kindly suggested. Additionally, we are thinking that a combination of the two areas (e.g., learning to meta optimize, or using L2O-discovered optimizer as base algorithms MetaOptimize, etc) might be promising research directions.

The reference on L2O for RL is particularly interesting. While the present manuscript focuses on supervised and continual learning (as more controlled settings), we plan future research extending MetaOptimize explicitly into RL.

---

Stability of MetaOptimize

How stable is your method?

All reported MetaOptimize experiments exhibit stable, consistent behavior across multiple random seeds. However, we observed instability in certain variants not included in the experiments (e.g., weight-wise or layer-wise step sizes in very deep networks). We explicitly mention these stability challenges in our "Limitations" section, suggesting directions for future research. Solving these underlying issues could significantly enhance MetaOptimize performance.

We thank Reviewer wz4K for thoughtful and valuable feedback. Below we provide clarifications, which we will also highlight in final version of the paper.

---

On Discount Factor γ:

Authors state γ=1 was used in stationary experiments, and performance meaningfully degrades for γ<0.999, making discounting somewhat contrived.

We clarify two important aspects:

• Our general framework naturally accommodates discount factors less than one. While stationary experiments favored γ≈1 due to implicit decay (discussed in the next bullet), having the flexibility to set γ<1 is beneficial for non-stationary tasks (e.g., continual learning scenarios, as shown in Section 7.2), where explicit discounting significantly improves performance. Thus, the framework’s generality allowing γ<1 is not contrived but genuinely beneficial for practical settings.

• The use of weight decay implicitly introduces discounting in meta-updates. To see why clearly, consider the base optimizer being SGD with weight decay parameter $\kappa$. The eligibility trace updates (under L approximation) become:

$$h_{t+1} = \gamma (1 - \alpha_t \kappa) h_t - \gamma\alpha_t \nabla^2 f_t(w_t) h_t - \alpha_t \nabla f_t(w_t).$$

Even if we set γ=1 and use a Hessian-free approximation (i.e., ignore the Hessian term), the trace $h_t$ still decays at rate $(1 - \alpha_t \kappa)$, exhibiting a discount-like effect. Thus, explicit discounting (γ<1) was unnecessary here primarily due to this implicit decay from weight decay.

Does backward formulation rely on γ<1? How does γ=1 impact derivation?

This is an important subtlety, and you correctly identified that our backward derivation (Eq. (6)) formally assumes γ<1 because of the scaling term $(1-\gamma)$ used for normalization. A straightforward workaround for γ=1 is simply removing this scaling factor $(1-\gamma)$ from the definition and all subsequent formulas. This slight adjustment makes the derivation fully valid and consistent for all γ≤1, matching precisely what we implemented and tested experimentally.

Your comment also highlights an interesting direction for future research. In RL and dynamic programming, the formulation and algorithms for the case γ=1 differ subtly, requiring additional reward-centering. Exploring analogous techniques for meta-optimization when γ=1 could be beneficial and is a promising avenue we will include in our future work discussion.

---

On Practical Value:

MetaOptimize lags behind well-tuned AdamW on TinyStories; thus, its practical value is unclear.

We acknowledge this observation and provide some clarification:

• The AdamW baseline used a carefully tuned step-size schedule through extensive search, specifically optimized for this task; while MetaOptimize involved no manual tuning. Thus, we did not expect scalar MetaOptimize to outperform such an extensively optimized schedule.

• In addition to avoiding manual hyperparameter tuning, MetaOptimize benefits from generality and applicability to generic meta-paramters beyond stepsizes. The biggest advantage however appear in continual learning (see experiment in Section 7.2), where tasks evolve over time and pre-defined stepsize schedules are ineffective.

• Crucially, we emphasize this paper’s primary contribution as foundational, laying the groundwork for a highly promising research direction. The greatest benefits of meta optimization methods are likely still ahead. We explicitly outline multiple clear pathways for future improvements (Section 9), highlighting that the current results are only the beginning. Thus, while initial performance is competitive rather than groundbreaking, the demonstrated potential and clear roadmap for future research represent the key strengths of this work.

Sensitivity to the base optimizer .., For example, base AdamW worked best on ImageNet while base Lion worked best on TinyStories.

The key point is that regardless of the base optimizer, applying MetaOptimize to learn the step sizes consistently outperforms the same base algorithm with best fixed step size. Note that the Lion optimizer tends to work better in training Transformers even when using fixed step sizes.

---

Scale of Experiments:

We agree scaling is important. Due to resource constraints, we focused on models <20M parameters, but included diverse architectures to show broad applicability. MetaOptimize is theoretically scale-agnostic and compatible with any first-order optimizer, so we expect it to extend naturally to larger models. Scaling up is a valuable next step and will be highlighted in future work.

---

Continual Learning:

Section 7.2 includes a continual CIFAR-100 experiment where MetaOptimize shows strong adaptation: step sizes increase after task switches to support quick adaptation, and decrease in early layers over time, reflecting stable feature reuse. These behaviors align well with continual learning needs, showcasing MetaOptimize’s potential in nonstationary settings.

We thank Reviewer Liw6 for their thoughtful and constructive comments. We carefully considered each suggestion and provide detailed clarifications below. We will apply the suggested improvements in the final version, which we believe significantly helps improve the clarity and presentation of the paper.

---

Reviewer’s Concern 1: Relevance of Eligibility-Trace-Style Approximation

“The technique proposed...follows the eligibility-trace-style approach, well-known in RL. Why do the authors believe this RL-inspired approach is relevant to approximating gradients, as these seem like very different tasks?”

The eligibility-trace-style approach is relevant and beneficial in our setting because it efficiently approximates the meta-gradient in a causal way: each term in the RHS of Eq. (5) directly appears in Eq. (4), though at the earliest feasible time it can be computed (ensuring causality). An alternative, would be direct gradient computation through finite differences which requires unrolling the optimization procedure far into the future at each iteration, resulting in prohibitive computational and numerical challenges; and also results in large delays in updates not acceptable in online and continual applications. In contrast, the eligibility-trace approach provides an instant and tractable approximation by summarizing gradient effects over time.

As noted in the footnote on page 2, our approximation becomes exact when the meta-stepsize tends to zero. For larger meta-stepsizes, approximation accuracy naturally decreases—a phenomenon also observed in RL. In RL, sophisticated techniques like Dutch traces have been developed to mitigate such inaccuracies (van Hasselt et al., 2014). Similarly, extending these advanced RL-inspired methods to our meta-optimization framework is an exciting direction we highlight explicitly as future work (Section 9).

---

Reviewer’s Concern 2: Comparison with Other Meta-learning Techniques (e.g., MADA)

“In reducing complexity, several approximations are used. It raises the question of comparing this technique explicitly with recent methods such as MADA (ICML 2024, Ozkara et al.).”

The MADA method builds upon the Hypergradient-descent approach of Baydin et al. (2018), updating meta-parameters based on immediate hypergradients. We have already thoroughly compared MetaOptimize against Hypergradient-descent in our experiments (referred to as the "gdtau baseline" in Section 5 and Appendix B). Crucially, we demonstrate theoretically and empirically that Hypergradient-descent corresponds exactly to the special case of MetaOptimize with the discount parameter $\gamma=0$. Thus, MADA inherently shares this limitation by considering only immediate-step effects, whereas our proposed method explicitly models long-term impacts via nonzero $\gamma$.

We appreciate your suggestion and will explicitly reference and clarify this comparison with MADA in the final version of the paper.

---

Reviewer’s Concern 3: Deterministic vs. Stochastic Meta-Optimizer

“In practice, stochastic optimizers are employed due to gradient intractability, whereas your paper presents a deterministic meta-optimizer. How would a stochastic version look, and was the deterministic version used in experiments?”

Thank you for highlighting this. We clarify that, despite the formal deterministic derivation in Section 4, our practical implementation is inherently stochastic. Specifically, the meta-update at each step (including the matrix $G_t$ computation) relies on gradients estimated using stochastic samples (such as mini-batches). Thus, in practice, MetaOptimize is already stochastic.

If desired, additional stochasticity could be introduced by employing further random samples (e.g., from a validation set) within each meta-update. We avoided this approach in our experiments to minimize sample complexity.

We will clarify the stochastic nature of our experimental implementation explicitly in the final paper revision.

---

Reviewer’s Concern 4: Complexity of Section 4 Derivations

“Derivations in Section 4 are complex. More explanations would greatly improve readability.”

Thank you for pointing out this readability issue. We will significantly enhance clarity in Section 4 by explicitly providing a concise outline of the derivations at the section's outset. Our goal is to clarify the key conceptual steps, making the mathematical reasoning easier to follow.

---

Reviewer’s Minor Comments

We are grateful for the reviewer’s meticulous attention to detail. We confirm that all minor suggestions (notation inconsistencies, typos, equation corrections) are valid and helpful. All indicated issues, including font inconsistencies, typographical errors, and minor mathematical clarifications, will be carefully fixed in the final version.