An Efficient Plugin Method for Metric Optimization of Black-Box Models

Main concerns:

1. The authors only compare empirically against calibration techniques. The proposed problem of finding a set of weights $\mathbf{w}$ to adapt a pre-trained model to optimize a black-box metric (which the authors refer to as query-access only metrics) is an instantiation of black-box optimization. As such, mature BBO techniques, including Bayesian optimization [R1], population-based search (e.g., evolution strategies) [R2], and even random search, should be included in the empirical comparisons.
2. The proposed line-search method has quite high complexity $\mathcal{O}(mn/\varepsilon)$, and scales linearly with number of classes $m$, and number of samples $n$, and inversely with search resolution $\varepsilon$. Techniques like BO and ES remove this dependence on $m$, $n$ and might be more suitable. They also do not require the pairwise comparison that is the basis of the line search.
3. Fundamentally and theoretically, what are the benefits of the author's proposed approach over these BBO techniques (including BO, ES)? Based on my understanding, the proposed technique has some theoretical underpinnings for linear-diagonal metrics, but as the authors emphasize that they focus on 'query-access only' metrics, it is not clear whether there are broader theoretical guarantees.
4. How is the reference class selected? The authors claim that the choice has little impact to the algorithm, but this is not justified. More specifically, how does the choice of reference class impact the optimization, since the line search is performed based on pairwise comparisons (against the reference class)?

[R1] Snoek, J., Larochelle, H. and Adams, R.P., 2012. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25.

[R2] Salimans, T., Ho, J., Chen, X., Sidor, S. and Sutskever, I., 2017. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864.

Limited novelty: The authors mention the “probing classifier” approach Hiranandani et al. (2021) that solves a, global linear system in order to find the weights which optimize a particular metric. CWPLUGIN is instead local in that it considers only pair-wise comparisons between classes. Is this the only difference between 'probing classifier' Hiranandani et al. (2021) and CWPLUGIN?
It is unclear if this difference constitutes a significant advancement, especially since the probing classifier already performs very well (e.g. in Figure 6, Probing in fact outperforms in terms of MCC and G-mean. Similarly, Figure 2 shows only marginal improvements in F-measure and accuracy compared to other methods, especially the probing classifier).

Key literature missing: The paper could benefit from citing some very important studies in the calibration literature that are relevant to the discussion, including:

1. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th ICML 2021.

2. Abbas, M., Zhou, Y., Ram, P., Baracaldo, N., Samulowitz, H., Salonidis, T., and Chen, T. Enhancing in-context learning via linear probe calibration. In Proceedings of The 27th AISTATS 2024.

3. Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. Prototypical calibration for few-shot learning of language models. ICLR 2023.

4. Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. ICLR 2024.

5. Zhongtao Jiang, Yuanzhe Zhang, Cao Liu, Jun Zhao, and Kang Liu. Generative calibration for in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of EMNLP 2023.

6. M. Shen, S. Das, K. Greenewald, P. Sattigeri, G. Wornell, and S. Ghosh. Thermometer: Towards universal calibration for large language models. ICML 2024.

These papers focus on calibration for LLMs and could also serve as valuable baselines, especially because the authors do studies on language tasks. Including them could strengthen the paper's comparative analysis and contextualize its contributions within the broader field of language model calibration.

Notably, Zhao et al. (2021) and Abbas et al. (2024) are especially pertinent, since these are also very simple post-hoc methods that use a handful of samples in the range comparable to the one used by the authors (i.e. on the order of tens or hundreds of examples). While these papers don't directly address the specific metrics used by the authors, they demonstrate high utility in terms of accuracy. It would be valuable to evaluate their performance on the metrics employed in this study, such as F-measure, MCC, and G-mean. Comparing CWPLUGIN against these methods would provide a more comprehensive assessment of its effectiveness.

Marginal improvement over baselines: Figure 2 shows only marginal improvements in F-measure (e.g. 0.579 vs 0.576) and accuracy (e.g. 0.619 vs 0.617) compared to other methods, especially the probing classifier. Moreover, in Figure 6, Probing in fact clearly outperforms CWPLUGIN in terms of MCC and G-mean.

Similarly, in Figure 3, on Language tasks, we observe marginal improvement over FullDirich baseline on most metrics except F-measure (e.g. 0.563 vs 0.562 on accuracy, 0.256 vs 0.255 on MCC).

Compared to the baselines, CWPLUGIN is the least performing method in terms of accuracy and MCC for lmemotions (figure 8) and one of the least performing methods on lmemotionsOOD (Figure 9). The results are perhaps good for applications where different evaluation metrics (e.g. F-measure) are more critical than mere predictive accuracy. However, I think even improvements on these metrics are marginal in most cases.

Figure 5 table ANLI with label noise results for G-mean do not match results in Figure 13 in the appendix. In Figure 13, Clean baseline always outperforms CWPLUGIN for all Validation Set sizes where as in Figure 5, Clean has lower mean of 0.528 as compared to 0.541 of CWPLUGIN.

Reproducibility: The code for reproducing the results has not been provided, which may raise concerns about the reproducibility of the findings. Making the code available would enhance transparency.

Minor comments: Line 263: typo: “…linear-diagonal metric Both results…” has a missing full-stop.

1. Novelty is limited in the case of CWPlugin. The method’s core is finding an optimal vector w to reweight a black-box model’s output. The novel contribution within the algorithm is the local search procedure over pairs of classes. The improvement here is rather incremental. As a more subtle point, the baselines used in Section 4 is inadequantely discussed in the related work subsection. Only probing is mentioned earlier in the paper. This results in a weak qualitative comparison with prior work.
2. A major section of the paper shows CWPlugin’s superior efficiency. This advantage is not demonstrated in the empirical section: there are no concrete runtime comparisons. Speedup from parallelization is also unverified.
3. The presentation of empirical results is not convincing. The tables adopt a specific validation set size, all different depending on the dataset but lacks justification. Sometimes the tables show 4 metrics, and sometimes only 2. This might indicate cherry-picking. The figures show more comprehensive results, but their presentation is poor. Legends are blocking the lines. Figure and axes titles are not formatted properly, and the color bands adds to the confusion. Based on the color bands, CWPlugin does not have an advantage in performance over the baselines due to overlap.
4. Further justification of why the datasets are chosen is necessary. Most of them also require proper citations. More importantly, the authors need to show these datasets as standard for previous works in the same field. For example, the income predicion dataset is uncommon in deep learning from a quick search. Overall, some of these aspects can be fixed and improve the quality of the paper.

Algorithm 1: The CWPLUGIN algorithm iterates over each class pair; however, the notation used is unclear. The symbol m denotes the number of classes but is also used as a class index, which creates confusion. Additionally, in line 3 of the algorithm, a single for-loop is shown, which does not accurately reflect the process of iterating over class pairs. Using a nested for-loop structure would better clarify this.

Class Size: The experiments were conducted on datasets with a relatively small number of classes. It would be beneficial to include results on datasets with a larger number of classes, such as adapting to specific metrics on CIFAR-100, TinyImageNet, ImageNet, to demonstrate the scalability of CWPLUGIN when m is high.

API Call Expense for Validation: The paper could discuss how the cost of API calls increases with the size of the validation dataset, which may be a limitation in scenarios where extensive validation data is needed to reach desired performance.

Suggested References: Two relevant papers are suggested to read and include:

• The first paper leverages relational information in label space to reweight prediction probabilities without requiring validation datasets: https://arxiv.org/abs/2307.12226.
• The second paper adapts black-box models by steering model outputs using a combination of tuned and untuned models: https://arxiv.org/abs/2401.08565.