Questions:

1. The relationship to learned optimization.

• "Learned optimization" can be viewed as a type of research in meta-learning (i.e., learning to learn). In this context, our work can be considered a form of learned optimization. However, learned optimizers are typically composed of black-box function approximators (such as neural networks) or adaptive hyperparameters tuners that are learned, as discussed in [1-3]. In contrast, our work focuses on learning to select among multiple categories of optimizers rather than learning a new black-box function or hyperparameters tuner. We have added relevant content in the related works.

[1] Andrychowicz, Marcin, et al. "Learning to learn by gradient descent by gradient descent." Advances in neural information processing systems 29 (2016).

[2] Daniel, Christian, Jonathan Taylor, and Sebastian Nowozin. "Learning step size controllers for robust neural network training." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 30. No. 1. 2016.

[3] Harrison, James, Luke Metz, and Jascha Sohl-Dickstein. "A closer look at learned optimization: Stability, robustness, and inductive biases." Advances in Neural Information Processing Systems 35 (2022): 3758-3773.

2. The relationship between SMBO technique and previous work on dynamic algorithm configuration (see Adriansen et al. JAIR’22).

• Thank you for the reference you provided. The problem we address falls under a type of "dynamic algorithm configuration", specifically referred to as "fine-grain tuning" in our draft. We have added relevant explanations in Section 3.2 to help better understand the content and significance of our research problem. Additionally, as pointed out in the reference, while the term "dynamic algorithm configuration" has been recently introduced, it encompasses a very broad range (cf. the first paragraph in Section 2.3). Previous work has explored "dynamic algorithm configuration" in contexts such as recursive algorithm selection, genetic algorithms, and learning rate control, whereas our focus is on the selection of optimizers. Additionally, while vanilla SMBO studies static algorithm configuration, IOMT applies this concept to dynamic algorithm selection.

3. The selection of hyperparameters $n_{ini}$ and $\tau$?

• In the experiments, we chose $n_{ini} = 50$ and $\tau = 25$ as the default settings for IOMT (cf. Section 5.1) to obtain sufficient training data for the surrogate model and enough switching instances across different datasets. We have added ablation experiments to analyze the impact of hyperparameter settings on the results to demonstrate the selected default setting for IOMT.

4. The choice of feature engineering method.

• We selected PCA as our feature engineering technique in IOMT because it is fast, effective, and easy to use. To further analyze the selection, we have conducted comparative experiments with other feature engineering methods (i.e., UMAP and random projection), and the results are presented in Table 10 (cf., Appendix D). We find that the simpler methods of PCA and random projection achieved higher test accuracy compared to the more complex UMAP method in the selected tasks (i.e., usps and mnist).

5. The selection of layers for surrogate model’s input.

• IOMT selects parameters from specific layers rather than from all layers to reduce computational costs (cf. the revised Paragraph 3 on Section 3.3). For partial fine-tuning, IOMT focuses exclusively on the trainable weights, such as the matrices A and B in LoRA. For full training, IOMT includes the classifier layer along with some sampled hidden layers as inputs. In our experiments, we found that selecting only the classifier layer already yields good results.

---

Results:

1. It would be beneficial to consider adversarial benchmarks like Imagenet-A to illustrate this advantage.

• We have conducted experiments on ImageNet-A and found that IOMT's dynamic optimizer selection leads to an improvement in accuracy (i.e., a top-1 accuracy increase of 2%), as shown in Table 5 (cf., Appendix C.2). Due to time constraints, we have not yet completed experiments combining more advanced techniques or applying them to state-of-the-art models, but we believe IOMT would yield similar improvements in those cases. We will continue our experiments and update the results in future versions.

2. The hyperparameter influence on the surrogate model’s quality (i.e., switching decisions).

• We have added additional independent experiments in Section 5.3 to explore the effects of various hyperparameter settings to switching results. Figures 10(b-d) demonstrate how variations in three key hyperparameters (i.e., the initial step size, the switch step size, and the number of PCA components) affect the experimental results. Although adjustments to these hyperparameters result in changes in test accuracy, a simple pattern emerges: larger initial step sizes, smaller switch step sizes, and fewer PCA components generally improve outcomes. This aligns with the conclusions presented in Section 5.1, which noted that the default hyperparameter settings yielded superior accuracy.

3. The comparison with just tuning hyperparameter for a single optimizer, i.e., the simple HPO baselines like tuning Learning rates with cosine annealing.

• We have included a baseline for hyperparameter optimization (e.g., tuning the learning rate with cosine annealing), with the results presented in Table 4 (cf., Appendix D). In addition, We have added a discussion in Section 4.2 on the advantages of mixing different types of optimizers, i.e., ``IOMT not only integrates the optimization strategies of different optimizers but may also yield a searched model that a single optimizer cannot achieve''.

---

Thank you for your valuable comments, which improved our work. We sincerely appreciate the time and effort you dedicated to reviewing our submission. In light of your advice, we have made the necessary improvements to our draft.

---

Method implementation:

The details on hyperparameter settings.

• We have added ablation studies in Section 5.3 to further investigate the impact of various hyperparameter settings in IOMT.

1. The influence of hyperparameter configurations for the initial weighted selection phase.

• We have conducted additional experiments in Section 5.3 to analyze the designs of the initial phase. First, as shown in Figure 10(a), our proposed weighted selection achieves greater stability in subsequent training compared to the random sampling used in the initial phase. In addition, the results of the ablation study shown in Figure 10(b) indicate that while a longer initial phase may result in some improvements in accuracy, the ongoing updates to the surrogate model can still yield reasonably good training outcomes even with a smaller $n_{ini}$.

2. The influence of the switch step size.

• We have added an ablation study regarding the switch step size in Section 5.3. The experimental results shown in Figure 10(b) indicate that frequently switching optimizers (i.e., using a smaller switch step size) typically achieve better accuracy performance.

3. The analysis on variance halving.

• We have added an ablation experiment on variance reduction in Section 5.3, specifically in the ``w/o halve'' entry of Table 7 in Appendix D. The experimental results show that variance reduction helps improve the final accuracy while lowering variance.

---

Presentation:

The presentation of the approach, while promising, could benefit from some improvements in clarity.

• We have revised the paper based on your suggestions to enhance clarity, including new visualizations, modified text descriptions, and clearer expressions.

1. Replace Figure 1 with an aggregated figure.

• We have updated Figure 1 according to your suggestions. Specifically, we have conducted multiple training sessions using various optimizers on four commonly analyzed optimization functions: Rosenbrock, Himmelblau, Griewank, and Ackley. Each optimizer was trained 200 times with different hyperparameters and random seeds. The results demonstrate that different optimizers behave differently at saddle points, even when various hyperparameter settings are applied.

2. The explanation in the introduction could be clarified (Lines 40–48).

• Thank you for your suggestions. We have revised the first paragraph of Section 1 to highlight the statement that "different optimizers result in varied weight configurations due to their unique decision strategies".

3. Figure 7 is somewhat unclear. More detailed labeling or annotations could clarify these distinctions.

• We have added new visual examples (i.e., Figure 7a) to help clarify Figure 7, demonstrating that IOMT can achieve optimization paths/model weights that differ from those obtained by a single optimizer. Furthermore, we have revised the corresponding text in Section 4.2 to improve clarity. Specifically, we have added a description of the model construction method in Figure 7a, which involves mixing different numbers of optimizers (i.e., altering the optimizer space of IOMT) and visualizing the differences in the final model parameters obtained after training for the same number of epochs (i.e., 100 epochs).

4. The author needs to clarify whether the authors are jointly optimizing for both optimizers and other hyperparameters, or if their focus is solely on optimizer selection (Lines 14-19).

• We have revised the text to emphasize the primary focus of our work. In particular, our research centers on switching between different types of optimizers, rather than selecting hyperparameters for a single optimizer.

---

We sincerely appreciate your dedicated time and effort in reviewing our submission. We have carefully revised the manuscript according to your useful comments. Below is our response to your suggestion.

---

W1&W2: Concerns on generalization to more hyper-parameters and search space.

A1: We have added experiments for IOMT and baselines for IOMT and baselines with an expanded hyperparameter space (cf. Paragraph 5 on Section 5.3 and the penultimate paragraph on Page 17). Under a broader range of hyperparameters, IOMT still achieves better test accuracy than baselines. Additionally, IOMT demonstrates stable performance even in an expanded search space, as shown in Figure 11.

---

W3: The definitions of w_p and w_d can be added to the Appendix. (Sec. 3.4)

A2: We have added the definitions of $\omega_p$ and $\omega_d$ in Appendix A. In the 3rd paragraph on Page 6, we provide that $\omega_p$ refers to performance-based metrics (i.e., accuracy), while $\omega _d$ pertains to distribution-based metrics, including LogME and Leep.

---

Q1: Potential overfitting of the proposed approach.

A3: The variance reduction described in Equation 7 on Page 6 can help alleviate overfitting. IOMT tends to select more stable optimizers (i.e., lower consideration for variance) at the late stage of training, which helps prevent overfitting to noise in the training data (similar to how a learning rate scheduler operates). Meanwhile, commonly used techniques for preventing overfitting—such as early stopping based on the validation set and learning rate schedulers—can also be applied to IOMT.

---

Thank you again for your comments to our work. Below are our responses to the additional experiments.

---

1. The changed HPs in Figure 11.

We have revised the corresponding descriptions and Figure 11 provide a clearer explanation of the additional hyperparameters. Specifically, we incrementally added the following hyperparameter search spaces in the experiments: (1) SGD weight decay, (2) SGDM momentum, (3) Adagrad weight decay, (4) RMSprop weight decay, (5) RMSprop alpha, and (6) Adam weight decay, in order to explore the impact of increasing the search space on the baselines and IOMT.

---

2. The performance of baseline approaches to varying numbers of HPs.

We have updated Figure 11 to include the performance of the baseline approaches (i.e., the results with the highest accuracy within the total search space). It can be observed that IOMT consistently outperforms the baseline test accuracy as the search space increases.

---

Thanks again for the clarification. Now I see the usefulness of the proposed approach, but I still have two concerns below.

1. The search space dimension is increased by 6, but only one hyperparameter is added for each optimizer, meaning that only two hyperparameters are considered for each algorithm. Therefore, the search space is still rather limited.

2. The baseline approach also performs robustly, implying that the added hyper-parameters are not quite difficult to optimize.

Thanks for your useful suggestions. We have carefully revised our draft based on your comments and addressed each point in detail below.

---

W1: The typos/grammatical errors.

A1: We have carefully revised the manuscript and corrected typographical and grammatical errors.

---

W2 & W7: Further verification on more datasets and tasks.

A2: We have conducted experiments on a more complex dataset (i.e., ImageNet-A), and included additional baselines (e.g., AdamW and CosineAnnealingLR). On the ImageNet-A dataset, IOMT achieves significantly higher accuracy than other methods, reaching 2% in top-1 accuracy (cf., Table 5 on Appendix C.2). Meanwhile, we have conducted additional ablation studies in Section 5.3. Due to time constraints, we will include more experiments on additional tasks and datasets in future versions.

---

W3: The definition of better model quality in page 1.

A3: The term "better model quality" in the draft refers to the improved performance results of the trained model, such as higher accuracy. We have added corresponding explanations in the text to aid understanding in Paragraph 1 of Page 2.

---

W4: The presentation of performance predicted by surrogate models is unclear: is it accuracy or something else?

A4: The term "performance" refers to the loss reduction achieved by various optimizers, as shown in Equation 2. We have revised the corresponding description in Section 1 for better clarity.

---

W5: The reason for using transferability assessment.

A5: We have refined the relevant statements in the 3rd paragraph on Page 6 to highlight the motivation of using transferability assessment. Specifically, the transferability between pre-trained models and the current task reflects the correlation between upstream and downstream tasks, which affects the ease of fine-tuning. For example, transferring a model pre-trained on the MNIST dataset to the SVHN dataset is easy because both datasets involve handwritten digits. In contrast, transferring between MNIST and CIFAR-10 is more challenging, as they have different characteristics.

---

W6: The presentation of Figure 1 is not clear.

A6: We have updated Figure 1 and added annotations for clarity.

---

W8: Comparison with more competitors.

A7: We have incorporated additional baselines (e.g., AdamW, NAdam, and CosineAnnealingLR). The experimental results are detailed in Table 4 (cf., Appendix C.2).

---

W9: This topic is not novel. Please read the paper about AutoML.

A8: We have added a corresponding explanation in the discussion of Section 4.1 to highlight the novelty of our work compared to current AutoML methods. Although IOMT is closely related to hyperparameter optimization, a key topic in AutoML, traditional methods generally use globally consistent static tuning or adaptive learning rate tuning, whereas we focus on fine-grain tuning of the optimizer during the training process.

---

W10: Use NAS to search for the best configuration for optimizers.

A9: We have added a hyperparameter tuning strategy (i.e., automatic learning rate tuning) and broadened the hyperparameter search space for searching the best configuration. The experimental results are in Table 4 and Figure 11 on Appendix. Compared to baselines with more optimally searched configurations, IOMT provides more robust outcomes and achieves superior test accuracy.

---

W11: More discussion of the generalization ability, robustness, and potential applications.

A10: We have added further discussion about IOMT’s generalization ability and potential applications through a new visual analysis (Figure 7a in Section 4.2). Additionally, we have conducted ablation studies to examine the impact of various hyperparameters and search spaces on the robustness of IOMT (cf. Section 5.3).

---

W12: The format of references is not correct.

A11: We have modified the citations in the draft which using the \citet{} format according to the ``Formatting Instructions for ICLR 2025 Conference Submissions''. We would like to know if the reviewer has identified any additional formatting issues for the references.

---

W13: Lacks theoretical analysis, ablation study, larger datasets, more tasks, larger networks, and visualization.

A12: We have added more experiments (cf. as in A2). Additionally, we have included new visual analyses in the revised draft (e.g., Figure 1 and Figure 7).

---

W14: Exploring the reasons behind the success of these techniques and providing intuitive explanations would contribute to the overall scientific contribution of the work.

A14: We have included a visual analysis in Section 4.2 that highlights the advantages of IOMT. This analysis demonstrates how IOMT can identify optimization paths that a single optimizer cannot achieve. Additionally, we have conducted new experiments in Section 5 to further investigate the rationale behind the design of IOMT.

---

Thanks for your useful suggestions. We have thoroughly revised our submission based on your comments. Below is our response to your suggestion.

---

W1 & Q1: The reason for IOMT’s performance improvement. (Sec. 4.2)

A1: Switching optimizers in IOMT can help find the optimization paths that a single optimizer cannot achieve (cf. Section 4.2). This expanded range of search paths results in improvements in model quality (i.e., higher accuracy). To clarify this, we have included new visualization examples in Section 4.2 Figure 7(a) to illustrate this situation better.

---

W2 & Q3: The impact and sensitivity of hyperparameter settings for IOMT.

A2: We have added ablation experiments for the hyperparameter settings (cf. Section 5.3). Figure 10(b-d) presents the experimental results, showing that variations in hyperparameters affect the performance of IOMT according to certain rules. Below is a summary of our findings.

• init step $n_{ini}$: Thanks to the continuous updating of the surrogate model during training in IOMT, even a small init step (i.e., $n_{ini}=20$) can obtain models with high accuracy. While a larger initial step, which provides more initial experience, may lead to a slight yet inconsistent improvement in test accuracy.

• switch step size $\tau$: A smaller step size facilitates quicker switching of the optimizer, which enhances accuracy in small datasets (e.g., usps). Conversely, a larger step size makes it more challenging to collect training data for the surrogate model, resulting in a longer switching cycle and greater variance in the results.

• PCA components number: Choosing a small number of PCA components (e.g., number=2) can already provide good performance in IOMT. In contrast, using a larger number of components may hinder the surrogate model's ability to learn effectively, leading to increased variance in test accuracy.

---

W3 & Q2: Unclear about the input parameters and time for PCA calculation.

A3: IOMT selects the key layers as input for PCA calculation, such as the trainable layers in partial fine-tuning and the classifier layer for full training (cf. Paragraph 2 on Page 5). We have revised the relevant content in Section 3.3 to clarify these points. Furthermore, PCA calculations are conducted at the start of each optimizer switch cycle (i.e., every $\tau$ iterations). We have updated Section 3.1 and Figure 3 for clarity.

---

Q4: The internal states inheritance when switching the optimizer.

A4: IOMT does not choose to retain internal states when switching optimizers. Our method emphasizes the "short-term" benefits that each optimizer can achieve given the current parameter state (cf. Section 3.3 Paragraph 4). Each time switching optimizers, IOMT treats that transition as the start of a new optimization process.

---

Q5: typos: "Iterleaving" -> "Interleaving."

A5: We have fixed it.

---