🏆 COPA: Comparing the Incomparable to Explore the Pareto Front

We thank the reviewer for the constructive and positive feedback. We are especially grateful for the kind words towards our work, acknowledging its importance and potential impact on modern ML. It is also encouraging to see the reviewer confirming the validity of our approach and derivations, as well as of the empirical validation of COPA.

The computational overhead of computing rankings for large model populations could be discussed more.

How does the method scale with very large numbers of models/objectives?

We agree, and we will add a paragraph discussing the overhead of COPA in detail for the camera-ready. Glossing over minor details, an implementation of COPA consists of sorting K arrays, each of them containing the performance of N models (see the implementation on the notebooks in the supplementary material). As a result, the overall time complexity of COPA is of $O(K N \log N)$, which is comparable to many light-weight preprocessing steps commonly used in the ML pipeline.

Some discussion of failure cases or limitations would be valuable.

Are there cases where the CDF transformation could be misleading?

We provide COPA, and thus the CDF, as a complement (not a replacement) of the original objectives as there is no need to discard them (i.e., the marginals). In fact, in figures 1-3, 5 and table 2, we plot the Pareto-front exploration in the original objective space to enable decision makers to perform intra-objective comparisons. Otherwise, we would lose vital information regarding the marginal information such as sudden phase-changes, as rightfully pointed out by reviewer w3Vj.

Regarding the limitations of the CDF, we can think of one main case where the transformation can be misleading: If the objectives turn out to be discrete rather than continuous (as assumed in line 55), then the resulting variable will no longer resemble a standard uniform one (as claimed in lines 180-184). We will include a paragraph stressing the need to meet our assumptions in the camera-ready version.

Additional ablation studies on the choice of p parameter could help inform practical usage.

While we already try to provide some guidance and intuition on the choice of $p$ to the practitioners, especially in the last paragraph before section 4 and the experiment in figure 3, we acknowledge that additional experimental results could help choosing $p$. Using the simple notebooks provided in the supplementary material, we will add extra results of the existing experiments using different values of $p$.

The connection to existing work in multi-criteria decision making could be expanded.

We will expand the existing discussion of related works. Besides MOO ML (i.e. estimation of the Pareto front) and multi-objective Bayesian Optimization, we will discuss existing works in multi-criteria decision making. We invite the reviewer to share any specific work they could have in mind and that we might have missed in the first version.

We appreciate the reviewer for their time and questions. We hope to have sorted out all existing questions and, if that were the case, we kindly ask to revisit the review if it feels appropriate. If there were further questions, we are happy to address them in the next phase on the rebuttal.

We thank the reviewer for their work, and we are happy to hear that the cases for which we apply COPA are interesting, and that our evaluation is new and appropriate. We hope the following helps the reviewer better understand our work.

This paper combine MOO and modern LLMs.

This paper is highly related to LLM evaluation and content moderation.

First, we want to stress that the scope of our work is multi-criteria evaluation in modern ML in general, being LLM selection one of our use cases in Section 5, which include LLM selection, domain generalization, fair ML, and AutoML benchmarking.

It seems that the author just gather the results from some existing models. Therefore, what is the contribution of this work?

Indeed, we gather publicly available evaluation data for all experiments except the synthetic and FairGrad ones (as stated in the appendix), which does not diminish the value of our work (at the end, these are our “datasets”) and only corroborates how broad and common of a task multi-criteria evaluation is, and thus the potential impact of COPA.

Our main contribution is a simple, yet general, approach to evaluate, compare and select ML models in terms of several (often) non-comparable objectives, by casting the problem to a probabilistic MOO problem (see lines 75-100 for further details). The simplicity of COPA is not a weakness, but a strength, and can potentially have significant impact in many areas of ML, as acknowledged by reviewer UMh1.

[…] Tchebycheff or the p-norm aggregation function has been proposed for years. [...] Is there new technical contribution in this paper?

While we hope to have cleared out the contributions of our work, let us remark that the proposed weighted $p$-norm in Eq. 12 is a novel and technical contribution. In lines 234-236 we discussed the differences between this norm and the usual weighted $p$-norm, which does not serve to intuitively map user preferences. To make this point more clear, we have reproduced figure 1 of the main paper using the usual weighted p-norm (see the new figure here). Furthermore, matching the Tchebycheff problem with $p=\infty$ is just one property of our norm that, however, does not hold for the regular weighted p-norm (as it can be seen here).

CelebA is not a LLM benchmark. What is the purpose of using CelebA as an introduction here?

CelebA is the dataset of one of the 5 total experiments we show in the main manuscript, covering many areas in ML (LLMs, fair ML, MTL, domain generalization, and AutoML). In our experiments CelebA is used to show how COPA enables practitioners to choose ML models that achieve a sensible fairness-accuracy trade-off.

DecodingTrust and Panacea references.

We appreciate the shared references, and we will add them to the camera-ready revision as related work.

Regarding DecodingTrust, we found it really interesting as it serves as yet-another-use-case for which the adoption of COPA can have a real impact, as the authors provide a set of objectives to evaluate LLMs and attempt to make them comparable (Appendix I.1), ultimately taking their average as score.

Given the similar format of the DecodingTrust and Open LLM Leaderboards, we have applied COPA to the formed leaderboard too, sorting the considered LLMs with different user-given preferences (we provide tables for three values of $p$). We will include the results as an additional experiment in the appendix. Among the most interesting take-aways, with COPA we can see that GPT-4 ranks among the least robust models in terms of DecodingTrust objectives (as it is the least fair LLM), while it is the 6th best model using the provided Overall score, as shown in their online leaderboard.

We hypothesize that the reviewer flagged our work for an ethics review by mistake. Otherwise, we would appreciate some explanation in this regard. We hope to have addressed all the concerns from the reviewer. If so, we would appreciate it if the reviewer could revisit their review to reflect these changes. We are happy to clarify any further questions in the next phase of the rebuttal period.

We thank the reviewer for their valuable feedback. There seems to be a misunderstanding with our work, which we believe to address below.

As long as we have a perfect Pareto front, practitioners can immediately find their own optimal choice

We politely disagree. As stated in the intro: Pareto fronts in high dimensional spaces are extremely difficult to visualize and navigate. In fact, for our use-case in Fig. 1 we had to summarize 6 performance scores to their average to depict the 487 Pareto-optimal models (out of 2148). Also, as we discuss in depth (e.g. in lines 50-54, 63-69, or 126-154), traversing the front over incomparable objectives is challenging: e.g., in Fig. 1, “Delta” maps half the preferences, $\alpha \in (0.5, 1)$, to a tiny region of the front (note the log-scale), where LLM performance is very low. The other baselines behave inversely similar, making it hard for practitioners to map their preferences to $\alpha$ (or $\omega$). In contrast, COPA maps the hypothetical (as we make clear in lines 96-100) practitioner’s preference of equal performance-cost compromise to $\alpha=1/2$. Of course, practitioners may require better performing models, which they can easily find by decreasing $alpha$ as shown in Fig 1-COPA, where there are another 486 Pareto-optimal models with different performance-cost trade-offs.

It is not proper to assume each there could be a weight between different objectives

We stress that weight-based preferences are still the de-facto approach in many MOO works [1-3]. While it is true that weights may not always be easy to interpret (see section 3.1.3 of [1]), we also remark that COPA overcomes this by making all objectives comparable first, for which we provide an interpretation of the weights in lines 237-251. Paraphrasing [1]: “Only normalizing the objective functions can one control the method to produce solutions of a desirable nature [...] Otherwise the role of the weighting coefficients may be greatly misleading” (as it is the case with the baselines in Fig. 1).

We acknowledge that there are other ways of expressing preferences—e.g., interactive methods, for which [1] devotes 80 pages—and we would love to explore them in future works.

[1] Nonlinear multiobjective optimization (1999)

[2] Smooth Tchebycheff scalarization for multi-objective optimization (2024)

[3] Revisiting scalarization in multi-task learning: A theoretical perspective (2023)

Users might like to maximize A+2B when C<0.5 whereas maximize A+B+C when C>=0.5 ...

We agree that users can often have complex preferences, and believe that COPA can readily handle many of them, including your example, where one can use COPA with the piece-wise criterion function described by the reviewer. Fig. 5 aims to illustrate how one can combine COPA with user constraints over original objectives.

Inspired by the reviewer, we have slightly adapted Fig. 1 to accommodate for such a case, see here.

While we agree with the reviewer that users can have complex preferences that COPA may not be able to handle, we believe that this is interesting but challenging future work that does not diminish the contributions of COPA.

Thus, we cannot assume the criterion function C must be differentiable and easy-to-be-optimized

There might be a misunderstanding: We do NOT require differentiability or easy-optimization for the criterion function, as we only need to evaluate for each model in Eq. 11 all the objectives. COPA is a multi-objective evaluation method, as stated in the intro, and we assume a given population of (already trained) models (lines 104-107).

With very small N, the accuracy of the estimated CDFs may degrade...

Our theoretical results on its variance in Prop. 3.1, the ablation study in App. A.1.1, and our experiments show that COPA is well behaved for already $N=15$ (Case 3). We will clarify that like, for any statistical estimator, COPA may suffer with extremely low $N$ values.

Rankings discard precise information about the relative distances between objective values

We provide COPA as a complement (not a replacement) of the original objectives as there is no need to discard them (i.e., the marginals). In fact, in figures 1-3, 5 and table 2, we plot the Pareto-front exploration in the original objective space to enable decision makers to perform intra-objective comparisons. We will stress this aspect in the camera-ready.

I do not think this trick is a sufficient contribution for a full paper.

We believe that, as highlighted by reviewer UMh1 in their review, the simplicity of our approach is a strength, which can be applied to many problems such as LLM selection, domain generalization, and AutoML benchmarking (see Section 5).

We hope to have clarified any concerns from the reviewer and, in this case, that they could reconsider their score. We are happy to solve more questions in the next round of the rebuttal.