HyperSHAP: Shapley Values and Interactions for Hyperparameter Importance

• To demonstrate the practical usefulness of HyperSHAP, its time cost should be clearly presented and compared.

  • As stated in the paper, Definition 2-4 require multiple HPO runs for argmax operations. Even with the use of an approximation, the time burden of HyperSHAP can still be significant.
  • The time burden of HyperSHAP raises concern about whether applying HyperSHAP before other HPO algorithms provides any real benefit.
  • More extensive and fair comparisons of HPO algorithms, with and without HyperSHAP, are necessary. Additionally, the time cost of HyperSHAP should be clearly reported.

• The paper does not include any comparisons with other methods specifically designed to measure hyperparameter importance or interactions.

• For HyperSHAP to be a reliable interpretability technique, its results on hyperparameter importance and interactions should match real benchmarks.

  • The experiments in Section 6.2 are quite limited in scope, analyzing only two types of toy optimizers on a single dataset.

• The experiments were limited in terms of search spaces and data modalities. Testing HyperSHAP on diverse configurations would improve its generalizability and robustness.
• The study would benefit from a comparison with other baseline explainable HPO techniques to assess HyperSHAP’s relative performance and computational efficiency.
• Calculating Shapley values and interaction indices can be computationally intensive. Including performance and scalability insights would be helpful, especially for real-world applications where computational cost is a concern.
• The paper is difficult to understand in some parts. For example, the Figure 2 is difficult to follow.

Even if the paper is well structured, it is difficult to follow :

• Much information that is required to understand what is going on is deferred to the appendix (e.g., how to read the plots, how the argmax are computed - there is not even a link to the corresponding appendix - for definitions 2 and 4). We should not have to resort to the Appendix to understand the results presented.
• l.421. All of a sudden, the authors use FSII, whereas it was barely mentioned earlier, and we do not know what it is.
• I find that the take-aways of the experiments are not clearly stated in Section 6
• Sometimes the link between the conclusions and the experiment's results is not clear. For instance
  • l.348 "For lower budgets, weight decay is not considered important." really? It seems equally important regardless of the number of evaluations in Figure 2.
  • l.350 "the optimizer leverages the strong positive interaction between weight decay and learning rate" There never seem to be any interaction between those two in Figure 2.

Although the framework identifies that hyperparameters can be of different importance, as well as their interaction, I am not convinced of its practical utility.

• The computational budget of this study is a serious concern for me. Indeed, evaluating SI involves a combinatorial number of model evaluations. The authors specify (in the appendix - it should be in the manuscript in my opinion) that they are evaluated with thousands of random evaluations, but it has a cost that is not taken into account e.g. in Section 6.3 in the x-axis reporting the number of evaluations. As a result, it is not sure that it effectively enhances the overall efficiency of HPO.
• What is the point of evaluating the weaknesses of a given optimizer if it implies to evaluate $\underset{\lambda}{\operatorname{argmax}} VAL_u(\lambda, D_i)$ l.360, i.e. basically solving the problem of HPO?

To me the main weakness is the limited experimental evaluation of the framework. It is evaluated on selected datasets within LCBench for neural networks (with rbv2 ranger benchmark in the appendix for random forests). The experimental evaluation works mainly as a way to showcase how the framework can be used and that it can be successfully used in at least some cases.

It would be quite valuable to see a more extensive evaluation, via a larger number of benchmarks with diverse sets of hyperparameters used - with some summary findings reported. Some of these could focus on fine-tuning settings (e.g. of large language models) as these are often of interest. Various pre-computed benchmarks exist so such significantly more rigorous evaluation may be manageable to conduct. For example, the PD1 benchmark could be a suitable example. Such more rigorous evaluation would help better establish the usefulness of the proposed methodology.

PD1 benchmark: Pre-trained Gaussian Processes for Bayesian Optimization. Wang et al., JMLR’24 https://arxiv.org/pdf/2109.08215