Thank you for your comment on the clarity of our paper. We will fix the typos and include a notation table in the camera-ready version. We would appreciate engagement in the rebuttal so we can address the reviewer's concerns.

We now respond to raised questions:

Comparison with Chierichetti et al. or Fotakis et al.

A direct experimental comparison with Chierichetti et al. and Fotakis et al. is not straightforward because our work addresses a fundamentally different problem. Our paper focuses on the choice modeling setting, using our BUCCHOI and DYPCHIP algorithms to learn a preference model from choice data—i.e., observing a single selected item from an assortment. Chierichetti et al. learn the central ranking from a dataset of observed top-k lists, which is a different and more information-rich signal than choice data. Fotakis et al. focus on the problem of aggregating incomplete and noisy rankings to find a consensus ranking; a distinct task from learning a generative model to predict single choices from assortments. Comparing predictive accuracy would require extending these models to handle choice data, a non-trivial task that our work addresses. To make this distinction clearer, we will expand our Related Work section to better contextualize our contributions.

Hyperparameters $p$ and beta were tuned on the test set (rather than a separate validation set), which is unfair against the MNL model

Our goal was to do a comprehensive study based on different choices of $p$ and $\beta$. To this end, we performed experiments for these different choices (through a grid search). Our goal was not to do tuning rather a comprehensive study. We can clarify this better in the final version. We would like to emphasize that even without tuning we get better test errors; the test error of our algorithm is at most 0.11 along all parameter choices while MNL test error is 0.168. Please see table 1 in the appendix.

Thank you for your time and review. We make sure to apply the suggested changes to make the paper more clear. Please see below for responses to your questions.

Question. In what practical use-case the p parameter would be set to a value larger than 1?

$p \\geq 0$ is the parameter for controlling the penalty per a pair of elements that are comparable in one top-k ranking but incomparable in the other.

Consider $\tau$ to be the top-k center. For instance, let $n=100$ and $k=10$, and consider the center $\tau=1, 2, 3, 4, 5,...10$. By increasing $p$ (e.g., $p>1$), the probability of any top-k ordering $\sigma$ with a small size of $\tau \cap \sigma$ will decrease. For instance, in $\sigma=11,12,13,14,...,20$ or $\sigma=12,13,...21$, or any other top-k with $\tau \cap \sigma=\emptyset$, there are ${10}\choose{2}$ elements that are comparable in $\tau$ and incomparable in $\sigma$, all of which incur the penalty of $p$. Meanwhile, in $\sigma=10, 9, 8,...1$, all pairs pay a penalty of $1$. Therefore, by increasing $p$, we decrease the probability of top-k's whose intersection with the top-k center is small.

The use case where $p > 1$ would correspond to cases where losing information regarding a relative order of two items is much worse than mis-ranking them, i.e., inclusion of an item in the top-k set is more critical than its exact position within that set.

A practical example is a product recommender system on an e-commerce website. Imagine a system that displays the "top 5 laptops for students." For a potential customer, it is crucial that the most relevant and suitable laptops appear on this initial list. If a highly appropriate laptop fails to make the top-5 cut entirely, the user may never discover it, leading to a poor user experience and a potentially lost sale. This large penalty for an item dropping out of the top-k list corresponds to a high value of $p$.

In contrast, if that same laptop is present in the top-5 list but is ranked at position #4 instead of its "true" rank of #2, the outcome is still largely positive. The user sees the item, can click to evaluate its specifications, and might still purchase it. The penalty for this internal mis-ranking (an inversion) is less severe.

Comment on the importance of the impartial culture model versus MNL:

In the impartial culture IC, the preference of each user is drawn independently and uniformly from the rankings of candidates. The Impartial Anonymous Culture (IAC) complements this model by using a uniform distribution over preference profiles (defined as tuples of rankings). Both of these models are primarily used in analysis of voting behaviors. In many applications, the simplicity of these models (in particular the assumption of uniform samples) is unrealistic therefore, the MNL model is used in a wider range of applications. We note that the IC model (uniform sampling) is a special case of the MNL and Mallows model. In MNL we obtain uniform distribution by setting all utility values equal to one and in Mallows model we obtain it by setting the discrepancy parameter $\beta$ equal to 0

The section 2 looks like it is almost copy pasted from the “Mallows models for top-k lists” paper: There are some similarities, although we state these definitions in a more general form and use inversions tables. In this section, we review the definition of Mallows models for top-k lists. Note that the main contribution of our paper is to use this model for modeling choice and choice probabilities.

We thank the reviewer for their helpful comments; we will use this feedback in preparing the camera ready version. Here is our response to some of the questions:

Re W1 (Computational complexity)

The sample complexity of $O(k·2^k)$ is only for a pre-processing step in PRIM (sampling algorithm), and generating samples after this step only takes polynomial time. We assume that $k$ is small, which is a common assumption due to practical setups in online platforms, and cognitive abilities of users for ranking items (see Chierchetti et al for more explanation). A common choice for $k$ is $10$ (with datasets collected as in thetoptens website) for which the preprocessing time is constant and hence generating samples takes $\Theta(\log n)$ (see Theorem 3.1). We also note that our algorithm is more efficient than the sampling algorithm of Chierichetti et al (even considering the pre-processing step).

Both in the preprocessing step of RIM and DypCHIP the $2^k$ term is an upper bound on the number of ``profiles’’. For $k$ close to $n$ the number of profiles becomes polynomial in $n$ and specifically for $k=n$ the number of profiles is only 1. For $k=n$ PRIM and RIM are the same algorithm since we will only have one profile.

Re W2 (Parameter learning from choice data)

Re W2a

We would like to clarify a few points about our learning algorithm BUCCHOI: (1) whether it is only applicable in active learning scenarios (2) how does the algorithm compare with prior works which use pairwise comparison data.

The crux of our active learning algorithm BuCCHOI is the FindTOP (Algorithm 4 in the appendix). Given an assortment $A$ which is repeatedly shown to the user and choice data collected, FindTOP finds: an element in the assortment which has the highest rank among other elements in the assortment w.r.t. $\tau^\*$ the center of the distribution. If such an element does not exist, i.e., none of the elements in the assortment are in the center (thus, they are all incomparable w.r.t. $\tau^*$ ) the algorithm returns none. A simple case is if the assortment $A=\\{i,j\\}$ has size two. In this case FindTOP outputs $i$ meaning $i>\_{\tau^\*} j$ or outputs $j$ meaning $j>\_{\tau^\*} i$ or None meaning $i\parallel_{\tau^\*} j$. Thus, we obtain full information about the relative order of $i$ and $j$.

If the assortment $A$ has size $\\ell$ we either infer $\\ell-1$ pairwise comparisons of the elements; or that all $\ell$ elements are incomparable and do not belong to the top-k center. For instance, assume given $A$ FindTOP returns $i\in A$ this means that $i>_{\tau^\*} j$ for any $j\\in A\setminus\\{i\\}$ ($\\ell-1$ relations) or it returns None meaning $i,j\in A\implies i\\parallel\_{\\tau^\*} j$.

FindTOP can be used either in a passive algorithm where the assortments are not selected by the algorithm or in an active algorithm. Clearly, if the assortments are not selected, we may only obtain some partial knowledge of the top-k center. It is not clear how to then expand this partial information to a top-ranking. We will add this question as future work in our conclusion and we will clarify that our methodology is an active algorithm in the introduction.

The reviewer is correct that BuCCHOI is only using pairwise comparison. But this is only for the simplicity of design. We could use FindTOP with various sets of assortments to find the center. For instance assume that the assortments must be all of size $\ell$. In this case we can use the following algorithm:

• Divide the n elements to non-intersecting assortments of size $\\ell$ (assuming that n is divisible by ell for simplicity):
• Run FindTop on these assortments, let $S = \\emptyset$
• • If FindTOP returns NONE then none of the elements in the assortment are in the center. Add them to set $S$.
• • If FindTOP returns $i$, $i$ belongs to the center, replace $i$ with an element $j$ in $S$ and run FindTOP on this new assortment.
• Repeat the above two steps until all elements of [n] are either in $S$ and for those which are not in $S$ all the pairwise orders are determined.

Note that this procedure concludes after $n/\\ell+2k$ assortment queries as with $n/\\ell+k$ calls we can get the top-k elements and with $k$ calls we can get the ordering of them. Similar to Theorem 5.1 we can prove that the sample complexity will be $O(\\ell^2\\log n)$.

The reason that we design BUCCHOi using pairwise comparisons is merely simplicity and many other algorithms similar to the above can be designed to infer the relative orders of all pairs of items.

We thank the reviewer for mentioning the existence of a broad body of works considering pairwise comparisons. When it comes to learning Mallows model center the most notable prior works are [Lu and Boutilier] and [Vitelli et al]. Both of these works are based on Maximum likelihood estimation and the EM method and only the convergence of algorithms are shown. These works differ from our work in the sense that they do not include any mathematical analysis of finite sample complexity bounds or theoretical guarantees on the error.

In our work by the use of concentration bounds we have provided a mathematical guarantee on the error of FindTOP and BuCChoi showing their finite sample complexity, This kind of mathematical guarantee does not exist in prior work which considers choice data or pairwise data and only exists when permutations or top-k samples are available or a linear order of some items are available. We thank the reviewer for pointing out this ambiguity and will add the above discussion to our paper.

• [Lu and Boutilier] Effective Sampling and Learning for Mallows Models with Pairwise-Preference Data, Tyler Lu and Craig Boutilier, JMLR 2014
• [Vitelli et al] Probabilistic Preference Learning with the Mallows Rank Model, Valeria Vitelli Oystein Sorensen, Marta Crispino, Elja Arjas, JMLR 2018

Re W2b

BuCCHOI is learning both the center and $k$. BuCCHOI (or any other algorithm which uses FindTOP such as the above) finds all pairwise ordering of the elements in [n] without knowing $k$. When these orderings are found, k is also found. Therefore, BuCCHOI also learns k. The reviewer is correct that we do not have an algorithm for learning the weights however we are not assuming that $\\vec{w}=1$. In order for our algorithm to work (Theorem 5.1), we just need that $\\beta> \\log 3/w\_{min}$. Learning weights requires additional analysis and hence we leave it as a future direction.

Re Q1

We are not sure what you are referring to here. We don’t have $w$ in line 143. Would you please clarify where you are referring to?

Re Q2

We considered the possibility of mixture models however, our algorithm is not directly applicable to mixture models. To handle a mixture of models, we first cluster the data to 2-5 clusters. Our thought for bypassing MNL mixture models was that we wanted to have an apple to apple comparison with our method which is not a mixture model. We note that, after doing the clustering the silhouette coefficient was always very low and we observed the lowest test error of our algorithm associated with no clustering. Therefore, we did not explore other possibilities for mixture models.

Re Q3

It is true that the Sushi dataset is top-10 and we can just focus on learning top-$k$ without assuming access to choice data, however the goal of our paper is to use choice models to learn top-$k$. Since this is a prominent dataset used in prior work and that we were not aware of any other dataset suitable for our method, we curated it to generate choices. We assume that the top-10 lists of the sushi data set are samples generated from a TopKMM and we learn its center. We have used these samples and our definition for choice for top-k lists (lines 231-233 of the submitted manuscript) to generate choice data. In particular, for a given (we chose $A$ randomly) assortment $A$ and sample $\tau$ we take the choice to be the highest element of $A$ with respect to $\tau$ if $A\cap \tau=\\emptyset$ we select uniformly at random an element of $A$.

I thank the authors for their detailed reply. I have a few follow-up comments and clarifications.

Re W1 (Sampling Tractability): My understanding is that sampling from the TopKGMM model is significantly less tractable than from the Mallows model. For instance, it is well known that one can efficiently sample full rankings from the Mallows distribution, but Theorem 3.1 suggests that it is not the case under TopKGMM. The “Profile-based” TopKGMM distribution appears to be a different (and arguably simplified) variant of the original problem. If the authors disagree with this assessment, a natural rebuttal would be to provide/point to an efficient sampling algorithm for the general TopKGMM.

I would like to note that this is not necessarily a fatal flaw, but potentially noteworthy since sampling is arguably one of the easiest tasks when it comes to ranking models. Various ranking models allows for efficient sampling at the very least; see references above. Given that even sampling appears intractable here, it becomes perhaps less surprising that other tasks, such as computing choice probabilities, may also require time exponential in k—in contrast to, say, the Mallows model.

Re W2 (Learning from general choice data)

Regarding learning from passive choice data, the authors may consider clarifying the following central question more directly: Given (arbitrary) choice data {(c_1, A_1), (c_2, A_2), ..., (c_T, A_T)}, how to infer the model parameters (e.g., central ranking and k) from the choice data? The authors seem to claim that they can do it (both in the paper and in the rebuttal). But what I can find is various simplified versions of this task. For example, FINDTOP focuses on a fixed assortment. BUCCHOI (i) assumes that the learner can choose assortments and (ii) focuses on pairwise comparison. Please feel free to clarify if I have overlooked any general inference result.

(On Related Work:)

The authors state that there is little work on error quantification for Mallows models and their variants. I would like to point to a few starters. For example, [Tang, Wenpin. "Mallows ranking models: maximum likelihood estimate and regeneration." International Conference on Machine Learning. PMLR, 2019] studies the statistical properties of the MLE for the Mallows model, and [Feng, Yifan, and Yuxuan Tang. "A Mallows-type Model for Preference Learning from (Ranked) Choices." Available at SSRN 4539900 (2023)] explores both statistical and sample complexity properties of a (arguably simpler) variant of Mallows model.

In the meantime, I wish to point out that even for pairwise comparison under the Mallows model, sample complexity guarantees could perhaps be easier to derive if the learner has the freedom to choose the assortments.

Q1: Resolved—thank you. Q2: This seems to be a schematic issue: any RUM is, in a technical sense, a mixture over rankings. In that regard, even MNL can be viewed as a mixture model (of ranking distributions). Q3: Two questions: (i) If the authors are generating single choice data from the SUSHI dataset, why not use the 10-SUSHI version with full rankings, which is more commonly used in the literature? (ii) If the parameter k is to be learned, what value is recovered from the SUSHI data?

Thank you for the supportive review. We appreciate the careful read and make sure to fix the typos for the final version. Please see below for responses to your questions:

Q1

The proposed methods, centered around the Generalized Top-k Mallows Model (TopKGMM), offer a robust framework for addressing practical problems involving user preferences, especially when those preferences are expressed as top-k lists rather than full rankings. For practical application, the process primarily involves two key algorithms: BUCCHOI and DYPCHIP. BUCCHOI is used first to learn the underlying central ranking ($\tau^\*$) and other model parameters from observed customer choices, which are typically recorded as (assortment, chosen_item) pairs. It achieves this by adaptively presenting specific assortments and inferring preferences based on selections. Once the model parameters are established, DYPCHIP efficiently calculates the choice probability for any given item within an assortment. This predictive capability is vital for businesses, enabling them to optimize product assortments, refine recommendation systems, and enhance advertising strategies.

Q2

We are not sure if we understand this question clearly. Please let us know if the following answers your question.

The main ranking acts as the fundamental "true" preference order for a user or a group of users. Rather than modeling preferences as a set of disconnected probabilities, the center provides a single, intuitive ranking that summarizes the core preference structure. This makes the model more interpretable; one can look at the central ranking to understand the typical preference of a customer population.

Thank you for your time and careful reading of our paper. We try our best to answer your questions and looking forward to constructive conversation. We will fix the typos and apply the suggestions for clarifying the paper.

[Q1-a] Why is Generalized Mallows sensible?

Generalized Mallows (for permuations) is introduced by Fligner & Verducci and it is a sensible extension because it more realistically captures user preferences by acknowledging that not all items are equally important. We develop this same extension to Chierichetti et al work on top-k lists .

[Q1-b] Why would you only have weights attributed to the higher element? Is this well motivated or does it simply make the math easier?

The weights are associated with the “rank” of items. The higher elements in the top-k have rank $1,2,\dots k$ and the lower elements all have rank $k+1$. This is consistent with the applications in which platforms have top-k rankings and the users are not exposed to the lower items.

[Q2] What is the sample complexity of doing the same things as you do in the paper for the Plackett-Luce model? In particular, the Plackett Luce model seems more amenable to top-k rankings, since its sampling algorithm exactly sampled in order of ranking. What is the main difference between Plackett Luce and Mallows, especially if in Plackett Luce once uses exponential weights?

Models based on Plackett-Luce, such as the Multinomial Logit (MNL) model are inherently different from Mallows models. The mallows model introduces a distribution based on the distance of a permutation (or a top-k list) from the center distribution (or center top-k list); this distance is defined in terms of the number of transpositions (swaps) to go from one permutation to another one. Plackett-Luce Model does not use a central ranking or a distance metric. Instead, it assigns an intrinsic positive score or utility to each item. The probability of a ranking is determined by a sequential choice process: the probability of picking an item first is its score divided by the sum of scores of all available items. The process repeats for the remaining items. Using exponential weights (e.g., as in the MNL formulation) does not change this underlying score-based mechanism.

Another main difference of PL/MNL model and Mallows models is in satisfying Independence of Irrelevant Alternatives (IIA) axiom which means the relative probability of choosing item $i$ over item $j$ is unaffected by the presence of a third item $\ell$. The Mallows model violates the IIA axiom while MNL satisfies this axiom. In Mallows model, the probability of preferring $i$ over $j$ depends on the entire central ranking $\tau$. This allows the Mallows model to capture more complex and realistic substitution effects among items, which is a key reason for its high predictive power.

We now respond to minor questions (MQ):

[MQ1] Why do you say that two items in a tau bar are incomparable as opposed to assuming they are a tie (mathematically)?

Could you please clarify this question? When saying i and j are incomparable we mean that the user has no preference of one over the other. As in previous work, e.g., Chierichetti et al, this is the terminology that is used. This shows a scenario in which users have no preference over less relevant items and they are ranked lowest for them.

[MQ2] Line 29: "High predictive Power of the Mallows model...": What do you mean by this? We mean that Mallows model when compared to MNL offers higher prediction accuracy (i.e., achieves lower test error) on several publicly available datasets. This is demonstrated by the work of Desir et al for the classic Mallows model and we show it here for top-k MM on Sushi data set.

[MQ3] Can you reference concrete examples where platforms show user specific top k relevant items and the k Is the same across all users? In many real-world applications, platforms have a fixed and limited space to display items. This is particularly true in online settings where, for example, a dedicated area on a website or an advertising box, e.g., on Google or Facebook, can only fit a predetermined number of choices. It also happens in the physical world, for example display areas of stores that display the latest arrivals however it has a fixed setup that would fit a predetermined number of items, e.g., stands in the store with a fixed number of shelves. Because of these spatial limitations, users are typically shown only the top-k most relevant items rather than an exhaustive list. This approach aligns well with user behavior, as people often express clear preferences for a limited number of favorite items and are indifferent toward the rest. Consequently, the preference data that can be collected from these platforms naturally takes the form of top-k lists instead of full rankings.

[MQ4] What is $\tau^\*$ in Equation 1? Do you mean $\tau'$?

Thanks for flagging this. There is a typo in the equation. We use $\\tau^\*$ to refer to the central ranking, and we changed Eq (1) to make it a general definition and not just with respect to $\tau^\*$. However, we seem to have forgotten to completely update it. $\\tau^\*$ should be updated to $\\tau’$ in the formula. We have updated our manuscript to reflect this.

We now responds to points made in Weaknesses:

[W0] What is the main different between Plackett Luce and Mallows, especially if in Plackett Luce once uses exponential weights?

Models based on Plackett-Luce, such as the Multinomial Logit (MNL) model are inherently different from Mallows models. The mallows model introduces a distribution based on the distance of a permutation (or a top-k list) from the center distribution (or center top-k list); this distance is defined in terms of the number of transpositions (swaps) to go from one permutation to another one. Plackett-Luce Model does not use a central ranking or a distance metric. Instead, it assigns an intrinsic positive score or utility to each item. The probability of a ranking is determined by a sequential choice process: the probability of picking an item first is its score divided by the sum of scores of all available items. The process repeats for the remaining items. Using exponential weights (e.g., as in the MNL formulation) does not change this underlying score-based mechanism.

Another main difference of PL/MNL model and mallows models is in satisfying Independence of Irrelevant Alternatives (IIA) axiom which means the relative probability of choosing item $i$ over item $j$ is unaffected by the presence of a third item $\ell$. The Mallows model violates the IIA axiom while MNL satisfies this axiom. In Mallows model, the probability of preferring $i$ over $j$ depends on the entire central ranking $\tau$. This allows the Mallows model to capture more complex and realistic substitution effects among items, which is a key reason for its high predictive power.

[W1] The experiments are down for the top-k mallows model not its generalized version and thus do not support the version introduced by the authors in this paper. This begs the question where one would use the weighted version

We extended the topKMM model to topKGMM model since our sampling and dynamic programming algorithms are applicable in the general setting. Our learning algorithm is also applicable to arbitrary weights as long as the condition of $\beta>\log 3/w_{\min}$ is satisfied (see theorem 5.1). The learning algorithm learns the center as well as the parameter $k$ and does not learn the weight vector $\vec{w}$; learning weights is a difficult problem and is left as a future direction. Various weight vectors (combination of 3s 2s and 1s) are chosen in synthetic experiments. For the experiments on sushi data we only used the topkMM model since we have no algorithm for learning the weights.

[W2] Figure 1 and the meaning of the test error is further not clear to me.

We have run the algorithms 10 times and the figure shows the mean and std of the test error.

[W3] I am not convinced that this paper solves an open question regarding a new RIM for top-k Mallows. My impression would be that RIM is a very nice and simple method, and something is RIM-like in so far as it is still simple, not because it uses insertions in some way

A key property of RIM is that it inserts the elements iteratively in the permutation (top-k list) by explicitly calculating the probabilities of insertion. This property is fundamental to the design of the dynamic programming algorithm for calculating choice probabilities. Our algorithm PRIM has a similar property: after sampling a profile inserts the element iteratively in the top-k by explicitly calculating the probabilities of insertion. This property is used in the design of DyPHIP.

[W4] Clarification request Thanks for flagging the areas that need clarification. We will make sure to address these in the final version.

Thank you for your questions, let us elaborate these two points a bit further.

1

We have $\tau^\* = 1 \\succ 2 \\succ 3 \\succ 4 \\succ \\{5,6,7,8\\}$ meaning that $1$ is the most preferred item, then $2$, then $3$, and then $4$, and $\\{5,6,7,8\\}$ are not ranked. When we count the number of comparable and incomparable pairs, we consider $i,j\in \tau\cup \tau^\*$. In fact, in the section under discussion, our goal is to show how we can rewrite the equation (1), where only elements in $\tau\cup \tau^\*$ matter. Since elements $7$ and $8$ are not ranked by either of $\tau$ and $\tau^\*$, they don’t contribute to $P_3$. Therefore, in the inversion table of 3, we are not comparing it with 7 and 8. The only pair that gets counted in $P_3$ is $4$ and we have $3\\succ_{\tau^\*} 4$ and $3\parallel_{\tau} 4$. We are happy to clarify this better in our camera ready version.

2

The Mallows model, analogous to the Gaussian distribution, is defined on the space of rankings, whereas the Gaussian distribution is defined on $\\mathbb{R}$. Both models posit a central value—a mean in the Gaussian context and a central ranking in the Mallows model—from which observed data are assumed to be distorted by noise. For instance, in the Gaussian case, a true scalar value of 3 might yield observations like $3.01$, $3.05$, or $2.95$, each with a probability derived from the Gaussian distribution. Similarly, in the Mallows model, a true top-$k$ ranking of $1 \\succ 2 \\succ 3 \\succ 4$ could result in observed rankings such as $2 \\succ 1 \\succ 3 \\succ 4$, $1 \\succ 2 \\succ 4 \\succ 3$, or $5 \\succ 1 \\succ 2 \\succ 3$, with probabilities determined by the Mallows distribution. In both scenarios, the probability decreases as the distance from the center increases.

In contrast, the Plackett–Luce model lacks a true central value; instead, each item is associated with a utility value. Consequently, in the Mallows model, the probability of an item appearing at a specific location in a ranking is dependent on other elements, while in the Plackett-Luce model, independence holds.

The Mallows model is applicable in any scenario where a true ranking can be assumed and observations are subject to noise. For example, a driver may have a preference over available routes from work to home (e.g., $A \\succ B \\succ C \\succ D$), but varying weather conditions, departure times, or interim errands might lead to observations of alternative rankings. Similarly, a customer's true ice cream preference ($A \\succ B \\succ C \\succ D$) might be obscured by situational preferences, resulting in observed rankings such as $B \\succ A \\succ C \\succ D$ or $A \\succ C \\succ B \\succ D$. This illustrates the concept of observed data being distorted from true values.

Our research, along with that of Desir et al., demonstrates that the Mallows model exhibits superior predictive power for the sushi dataset, a widely used benchmark for ranking data.

Please let us know if this sufficiently addresses your concerns, and if we can provide any further clarification.