COPA: Comparing the incomparable in multi-objective model evaluation

We appreciate the feedback and comments by the reviewer, and are happy to hear that our work is well-written and easy-to-follow for people unfamiliar with multi-objective optimization, which is ultimately one of our goals. Let us then attempt to address those concerns raised by the reviewer:

This paper presents a comprehensive method, supported by many experiments and a good presentation. However, from a research perspective, its contribution is limited.

We can see how our work can seem limited from a technical perspective (yet we have technical contributions), but we disagree that our research contributions are limited. Not all works need to propose new methods nor complex derivations. Instead, our work highlights an existing, prevalent, and underexplored problem in ML model evaluation by framing as a multi-objective problem, and it provides a simple and intuitive approach to address this problem. COPA has been carefully designed and its simplicity makes it likely to have a real and significant impact, as the reviewer acknowledged in their review.

Moreover, we show how to adapt our assumed setting of model selection to more unconventional and surprising scenarios. For example, we show how objective incomparability can bias the conclusions drawn in research works in areas such as multitask learning and domain generalization (section 5.3). While this is clear once it is explained, it is not direct nor obvious, as otherwise existing works would have attempted to address it (however, recent works have started to partially tackle the issue with rank averages, as we properly acknowledge in lines 269-273).

I think that the proposed normalization method removes excessive semantic information from the objective values. I think that considering proper Pareto optimality [1,2] is more appropriate.

We fail to see the concern of the reviewer, as rankings are order-preserving and therefore preserve Pareto optimal points (D4, line 153). Moreover, note that we do not ask the user to drop the semantic information of each objective, as it is valuable to make intra-objective comparisons (or constraints, as in Figure 5). What we ask the user is to use relative rankings to perform inter-objective comparisons, as we can make more-informed comparisons across objectives that way. This is also why almost all figures but Figure 4 in the manuscript are plotted using the original objectives, and not their transformed values.

In other words, certain solutions are unsuitable and should be excluded. After removing these solutions, applying norm may be enough.

We similarly fail to see this point. Without applying any transformation, the objective vector of each model should not depend on the population and, since Eq. 11 guarantees to find Pareto-optimal solutions (lines 242-248), the solution found should be the same regardless of whether we restrict our space of solutions to Pareto-optimal models or not. We have extensively shown in our paper that this approach (which we term “naive”) does not work well in practice (e.g. in Figure 1).

What is the computational complexity of the normalization method?

The proposed normalization has the same complexity as sorting an array and dividing by its length, that is, O(n log n).

When a new $h$ is included in $\mathcal{H}$, the relative rankings may change. The paper does not discuss how to efficiently update the CDF approximation without recalculating all rankings. I think this might be interesting.

We appreciate the feedback and agree that it can be of interest if the number of models is excessively large. In said scenario, one can update the ranking by finding the position of the new point in a sorted array in O(log n), and then adding a constant 1 to all entries after this position (worst case complexity of O(n), yet it can be practically constant with vectorized operations).

We hope to have addressed all concerns and remain at the disposal of the reviewer had they had more questions to come. We also appreciate the suggestions and spotted typos which we will implement in the next revision of the manuscript, and we hope the reviewer can reconsider the re-evaluation of our work.

We thank the reviewer for their feedback and positive comments. We are especially happy to hear that the paper was clear, well organized and meticulous, and that we successfully demonstrated all our claims throughout the discussion and experiments conducted in the manuscript. Let us then address the concerns shared by the reviewer.

We recommend creating a dedicated "Limitations" section, even if concise, to consolidate these discussions.

We do agree with the reviewer, and we will add a limitations section in the revised version of the manuscript, gathering and summarizing the parts critical to make COPA work in practice (e.g. the number of samples, or the need for continuous random variables).

The paper does not provide sufficient information on the computer resources.

We apologize that this is the case, and will provide more details in the next revision. To be more specific, as COPA does not need to train any model, but to evaluate them, everything it needs is to store a matrix of N models and K metrics, sort each column, and compute a norm, which can be done in O(n log n).

In practice, all experiments were done in a single core of a mid-range laptop. This also includes the synthetic experiments and the training of the fairness use-case, which we had to explicitly reproduce from the original paper.

How does COPA perform if the population of models (N) is very small and the CDF approximation via ranking is less reliable?

In App. A.1. we ablated COPA as we vary N, and found it to be quite reliable. Naturally, the estimation will worsen as we reduce N (12 is the lowest we tested), but we encourage the reviewer to reproduce Figure 9 several times by themselves with the provided supplementary material (synthetic-case.ipynb) to gain a hands-on feeling.

What is considered a "sufficiently large" N for COPA to be effective in practice?

Sufficiently large is rather subjective, but we note in the inset figure after Prop. 3.1 that for a small sample size of $N=10$, the maximum variance of the estimator has been reduced from 0.25 to approximately 0.025, which is quite significant. In the experiments, the minimum we have experimented with is $N=15$ in section 5.4, which works qualitatively well as the plots lead to similar conclusions as the original figures with scaled performance.

What are the specific implications if objectives are discrete rather than continuous? How does the approximation of the CDF via ranking handle this, particularly regarding the "uniform distribution" property and the potential for many ties?

This is unfortunately a difficult question to answer, as the probability integral transform only holds for continuous random variables and the current version of COPA heavily relies on this result. Speculatively, COPA should be adapted by finding a transformation for discrete random variables which turns them into the same common language, as we argue in lines 178-182 for the continuous case. A good starting point could be the work by Gery Geenens in discrete copulas [1], but we defer this direction for future work.

[1] Geenens, G. (2020). Copula modeling for discrete random vectors. Dependence Modeling, 8(1), 417-440.

We hope to have successfully addressed all concerns by the reviewer, and to re-evaluate and champion our work during the rebuttal period if that were the case. If there were further questions, we will be happy to answer them.

We appreciate the extensive feedback by the reviewer, and we are looking forward to clarifying any concern during the rebuttal period that can encourage the reviewer to re-evaluate our work.

Goal of our work:

First and foremost, we believe there is a misunderstanding regarding the premise of our work based on the following snippets from the review:

The premise of the paper is to define this function for them. If this were possible, then MO would be reducible to a single objective, which it is clearly not.

However, it is well known that multi-objective optimization is not reducible and any attempt to do so will create an implicit biased order that effectively hides points from the DM.

In general, the DM's internal "total ordering" function is not knowable.

It is still a multi-objective problem for the user wrt $\omega$ and $p$.

This paper is a way to order objectives according to a certain criterion that works in "these cases".

which led to the following final statement by the reviewer which we do respectfully but strongly disagree with, and which also sadden us to read regarding our work:

So ultimately the paper provides false hope and a specific biased view on problems that will sometimes be applicable and sometimes not with no warning to the DM for when the assumptions of the approach are violated.

The goal of our approach is not to reduce the MO problem to a scalar problem, but rather to assist the user into making informed and meaningful decisions about the model they select in the MO framework. This is to say that we attempt to normalize all objectives under a common framework (Eq. 5) and a tunable criterion function with interpretable parameters (Eq. 11), so that the user intentions can be clearly reflected and mapped into the Pareto front. Of course, the parameters $\omega$ and $p$ are an imperfect approximation of what the DM would want, all models are approximations. But we think COPA is a step forward into making an approximation (even if as simple as COPA is) actionable in a principled way.

We did try to make this clear along the paper, e.g., in lines 72-74 where we say: “COPA is a novel approach to allow practitioners to meaningfully navigate the Pareto front, and thus compare and select models that reflect their preferences”, but we will further clarify the scope of our work in the camera-ready.

Questions and comments:

Let us now attempt to clarify other parts that may not be clear from our work based on the provided review:

but this is implicitly requiring the DM to define a ranking, possibly missing a good solution that is filtered out/ranked out of visibility under the new metrics.

This is prevented thanks to desideratum D4 “order-preserving” (line 153). Ranks preserve the ordering of each objective, and therefore Pareto-optimal points wrt the original objectives. Stressing the point above, we do not require the user to fix $\omega$ and $p$ (and thus the ordering), they are free to explore as many values as they want.

They are assuming that the ordering of the objectives induces comparability.

We do not assume such a thing. It is well-known that a 1D continuous random variable can always be transformed to a standard uniform by applying its cumulative function (lines 171-172) [1, Theorem 2.1.10], which we approximate by taking rankings in Eq. 9. This fact is also known as the universality of the uniform, and we provide an interpretation of why objectives are indeed comparable after this transformation in lines 178-182, as now all values refer to their performance relative to the same population.

[1] George Casella and Roger L Berger. Statistical inference. Cengage Learning, 2021.

They mention this fact [other methods attempt to reduce MO to a scalar problem] as a weakness in related work sections.

We fail to see where we mention what the reviewer refers to. What we say in the related work is that prior approaches fail to address objective incomparability (line 265), and that others instead define ad-hoc approaches to address it in their specific problem instance (line 268). We are happy to further clarify this in the next revision.

They do this, I think, by assuming that the objective values are, in a sense, nearly uniformly spread throughout the min/max values for a given objective.

We do not make this assumption anywhere. Again, transforming a continuous random variable using its own cumulative distribution function will turn the variable into a standard uniform.

Essentially, they are using the “median” statistic in a sense (as they mention).

No, what we say in lines 181-182 is that, after transforming the objectives using Eq. 8, a value of $u = 1/2$ semantically corresponds to the median value for all objectives. In other words, all values of $u$ refer to the same quantity for all objectives, and are therefore comparable across them.

the objectives could be in tight clusters for the first objective, where the clusters have dramatically different values for a given cluster. [...] The method should be extended to handle clustering of values where the ranking may not be meaningful within the cluster of values.

This could happen, and we appreciate the feedback and suggestions, which we will consider in future work. However, please note that we do not ask the user to discard nor disregard the original objective values: indeed, they are useful to perform intra-objective comparisons. If the original objective distribution were multimodal, the user should be able to compare the original metrics and see a large discrepancy between values, which can then be further analyzed. This is also why almost all figures but Figure 4 in the manuscript are plotted using the original objectives, and not their transformed values.

They should provide examples where their system fails to highlight its weaknesses, along with its strengths.

We agree with the reviewer, and will gather and extend our assumptions, insights and limitations into a common “Limitations” section in the next revision.

This is likely renumbering the indices of that is assumed not described which was confusing.

We are sorry for the confusion, and indeed the expression $y_{R(1)} \leq y_{R(2)} \leq \dots \leq y_{R(N)}$ is a typo, which we wrote in an attempt to clarify what are order statistics. This will be easily fixed for the camera-ready and we thank the reviewer for noticing it.

Here, $R(i)$ are order statistics and the reviewer can think of them simply as rankings. In standard notation, one can write $y_{(1)} \leq y_{(2)} \leq \dots \leq y_{(N)}$, but since our notation refers to the inverse mapping, there is no clean way of writing such expression. We will fix this line by explicitly stating what $R(i)$ refers to: the new position of the i-th sample after reordering the array of objectives.

Some guidance should be given how the DM might decide when there are more than a few objectives.

We agree and will add a paragraph to better guide the DM. For example, in section 5.2 we followed a strategy that first groups objectives by what quality they measure (e.g., performance vs. cost) and then we assigned a weight $\alpha$ to carbon footprint, and the same weight of $(1 - \alpha) / 6$ to the rest of metrics. This hierarchical strategy can easily be applied to other scenarios.

We hope to have clarified all the concerns of the reviewer, especially those concerning the scope and overall goal of our work. If this were the case, we would truly appreciate a reevaluation of our work with this newly-adquired perspective.

Dear Reviewer fEWz, given the authors' response, please raise any remaining questions and/or concerns in a timely fashion so the authors have a chance to reply. Thanks!

We thank the reviewer for their comments. We are particularly pleased to hear from the reviewer that COPA is an elegant, grounded and simple solution, and that the paper is well organized. We now address the raised concerns:

if you add more models, the whole rank would change and need to recompute everything.

This is true, but note that computing the rank is a cheap computation of O(n log n). If this were a concern, one could still update the rankings by finding the position of the new model in a sorted list in O(log n), and then adding a 1 to all the values after that entry in O(n) at worst. Note that in practice one can speed up this linear factor further via vectorization.

the results are "relative", not an absolute score like Elo scores which hugely depends on the small circle of models.

The results are indeed relative to the population $\mathcal{H}$ in play, all of them. That is to say, their rank does not depend on a subset of “similar” models, which is what we understand when the reviewer refers to “the small circle of models”.

when the metrics are discretized, it is hard to rank them or quantify them. Many models would be on-par. E.g. sense of humor: Yes/No.

True, but we assume continuous metrics in our problem statement (lines 87-88), so your concern is already addressed. In the case of discrete variables, and hence in the presence of ties, Eq. 9 gives them the same (minimum) rank. For example, assuming we have ties for a continuous array of metrics [0.2, 0.5, 0.5, 4] would get ranks [0, 1, 1, 3]. This is not an issue for our methodology.

How sensitive are the final model selections to the composition of H?

We provide in Prop. 3.1 key statistical properties of the rank estimator (and therefore of the final model selection) with respect to the model population $\mathcal{H}$. Moreover, provided i.i.d. samples the given estimator converges uniformly to the empirical distribution (see the Dvoretzky–Kiefer–Wolfowitz inequality). Finally, we explicitly evaluate the sensitivity of COPA with respect to the model population in App. A.1 of the manuscript, where we find it robust even as we re-sample (and decrease) the number of samples in $\mathcal{H}$.

Does this volatility undermine the goal of finding a "globally" good trade-off, and are there ways to mitigate this, perhaps by using historical data or defining a more stable reference population?

This is an interesting question that deserves a follow up! Extending COPA to cases with historical data or to online settings are exciting (but non trivial) venues of future work. We need to establish COPA for global, non-online setting first (as we did in this paper).

For a large number of objectives K [...] setting the weight vector ω is non-trivial. Could you elaborate on practical strategies for users in such scenarios?

We provide in lines 227-235 an interpretation of the weights under our framework with the aim of assisting the decision maker choosing $\omega$. In a scenario as the one described by the reviewer, one strategy to simplify the choice of $\omega$ can be to group objectives by their overall goal (e.g. some might measure different types of robustness, while others could measure performance) and provide weights to the groups directly. This is what we actually do in, e.g., section 5.2, when we assign a weight $\alpha$ to carbon footprint, and of $(1 - \alpha) / 6$ to the rest of metrics.

We hope to have addressed the concerns shared by the reviewer, and would appreciate it if the reviewer could update their score accordingly as well as share any remaining comments they may have.

We thank the reviewer for their feedback and comments. Let us next attempt to resolve all the concerns raised and, if we correctly address them, we would appreciate it if the reviewer could reconsider their evaluation of our work.

Much of the approach is akin to "How to Make Multi-Objective Evolutionary Algorithms Invariant to Monotonically Increasing Transformation of Objective Functions" Yamada et al 2024.

We appreciate the reference by the reviewer, which we were unaware of and will add to the next revision of the manuscript. However, while we see the similarities, we respectfully disagree in that both approaches are as similar as to disregard our methodological contributions, i.e., the paper cannot be rejected in these terms.

First, we compute the empirical CDF of the marginal (i.e. wrt. all samples), while the work by Yamada et al. computes that of the non-dominated solutions. This leads to two different estimators, one being computed on a subset of samples of the other. Whether it is preferable to define Eq. (1) wrt. all or only Pareto-optimal solutions is an interesting question beyond the scope of this work. Second, our criterion function in Eq. (11) subsumes those considered by Yamada et al., as they only consider $p=\infty$ ($w_1$ in their notation). Note that $w_2$ in their work corresponds to the usual weighted p-norm and not to Eq. (11), which leads to significantly different results, see Figure 10 in our appendix.

The contribution of this work is largely applying this technique to model comparison, insight of population selection in evolutionary algorithms.

Disregarding the technical differences mentioned above, we fail to see where the insights mentioned above can be found in the reference provided and would appreciate pointers. Moreover, we would like to remind the reviewer that not all contributions of a work need to be technical. Our work highlights an existing, prevalent, and underexplored problem in multi-objective ML model evaluation and provides a simple and intuitive approach to address it, as acknowledged by the reviewer.

This is, on its own, an important contribution but, more remarkably, we show how to adapt our assumed setting to more unconventional scenarios where the adoption of COPA can have a real and significant impact, as acknowledged by reviewer Wm5j. For example, we show how objective incomparability can bias the conclusions drawn in research works in areas such as multitask learning and domain generalization (section 5.3).

The baselines are limited. Are there other normalization techniques to consider?

As mentioned above, this is an underexplored problem. There are other simple but general normalization functions that we are aware of (e.g. the inter-quantile normalization) and will be happy to add them and run experiments if the reviewer can provide a concrete recommendation, as well as any specific normalization technique the reviewer could point to.

More often, we find that authors come up with ad-hoc normalization techniques to try and avoid issues arising from the incomparability of metrics, as we point out in the paper, see lines 267-271 or App. A.6 where we explicitly give all the normalization functions employed in DecodingTrust.

How should a user consider choosing p?

We try to provide advice and intuition on how to choose p in lines 236-241, 248-252 as well as in section 5.1. In short, larger values of p focus on maximizing individual metric performance (and thus robustness in the min-max sense), while lower ones focus on aggregated performance (e.g. a value of one corresponds to a weighted average performance). We recommend initially choosing p based on the user expected robustness, and explore other solutions by nudging this initial value until satisfied, since evaluating COPA after normalizing once is linear in the number of models.

L294: "finding a proper ω could prove challenging". Can you elaborate on that?

Of course, let us clarify this statement. With $p=\infty$, we gain a finer control with $\omega$ on the solution found, as this formulation can find any Pareto-optimal solution (lines 246-247). A good analogy of this is that we can think of the solutions found with $p=\infty$ as those found by projecting a ray from the origin towards $\omega$. Therefore, if we have many objectives ($K$ is large) and we are interested in a really particular solution, tuning $\omega$ to find said solution can be time-consuming as we would need to carefully tune each of the $K$ components of $\omega$.

We hope to have clarified all the concerns of the reviewer, especially those concerning the contributions our work. If this were the case, we would truly appreciate it if the reviewer could update their score accordingly. Thanks again for the feedback, and we remain at the reviewer’s disposal in case there were further questions.