Enhancing Tail Performance in Extreme Classifiers by Label Variance Reduction

We thank the reviewer for their valuable feedback.

Can the teacher model be adapted to the current task? How does its performance compare?

The teacher model is adapted to the current task, the Siamese model is trained on the same dataset on which the classifier-based methods were trained. The teacher model (Siamese network) is much better than the student model (classifier) on the tail (indicated by superior PSP Coverage), while it lags behind significantly in Precision. Using the superior performance of the teacher on the tail, LEVER improves the OvA classifier’s tail performance. We have updated the draft to include a comparison between teacher and student in Table 4 in the main paper.

What are the differences between directly using a better model for distillation and utilizing the results of other models as auxiliary information in this paper?

We request the reviewer to clarify the question. From what we understand, the reviewer is asking the effect of using a better model for distillation. If so, it is the case that a better teacher model results in better performance of classifiers when combined with LEVER. Table 13 in the supplementary shows the effect of using different teachers. We experiment with 3 teachers: Astec, MiniLM 3-layer, and 6-layer DistilBERT.However, it is important to note that even our best teacher, NGAME 6-layer DistilBert, performs much inferior to classifier-based methods on precision.

Readability of paper not strong and uncomfortable formatting. Full text not filling 9 pages. More discussion around train time and teacher model performance.

We apologize for the concerns around formatting. We have made the following changes: (i): Abstract has been made concise, (ii): Empty space around Section 2 related work and equations has been cut down, (iii): We have devoted the additional space obtained to discuss the train time overhead of using LEVER (Table 3) and Comparison with the teacher model (Table 4)

Limited novelty: the innovation lies in using a teacher model to predict probabilities. From a personal perspective, the innovation lies in using a teacher model to predict probabilities From the perspective of innovative design, it is not convincing.

While the approach might be simple, we believe the concept of label variance we propose is novel. Additionally, we would like to highlight the difference between our setup and conventional distillation, which usually relies on a teacher model that is uniformly better than the student. In our work, we show that a teacher model that excels on a subpopulation (tail-labels) can also be leveraged to improve the performance of OvA classifiers for XMC tasks. Further, the simplicity of the approach should rather be viewed as a strength as it allows for easy and effective combinations with multiple SOTA classifier-based methods.

Why does the Siamese model provide accurate estimates of $p_x$?

Recent studies have shown that Siamese Networks, when used as input encoders, exhibit impressive performance on tail labels (Dahiya et al., 2021a; 2022; Jain et al., 2023). This success can be attributed to the ability of Siamese encoders to learn correlations by utilizing label-side features. Further, we consume only high confidence estimates of $p_x$ from Siamese teacher models: “For each label l, the τ nearest points from the pool are selected and added as ground truth.”

---

We hope this rebuttal clarifies the concerns you had about the paper. We would be happy to discuss any further questions about the work, and would really appreciate an appropriate increase in the score if the reviewer’s concerns are adequately addressed to facilitate the acceptance of the paper.

We thank the reviewer for their valuable feedback. We request the reviewer to clarify the ethical concerns with our submission as we see our paper has been flagged for ethics review.

Below we clarify some of the concerns.

Calibration concerns

Thank you for bringing this paper [1] to our notice. Since our loss is shift-invariant, the scores produced by the model are not calibrated, as pointed out in [1]. If $s_x =z_l^{T}z_x$ is the score output by the Siamese model for a label $l$ (with embedding $z_l$) and document $x$ with embedding $z_x$ this would mean that $s_x$ is not calibrated, we agree. However as indicated in theorem 2, we make use of probabilities $p_x = 1/(1 + e^{-(ms_x + c)})$ where $m,c$ are hyper-parameters for training OvA classifiers. Note that this is in line with the post-hoc calibration strategies explored in [2], where a model is learned and then a parametrized Sigmoid function is fit to learn the relevance probabilities, i.e., given an uncalibrated score $s_x$ we learn $p_x$ by fitting the sigmoid function $p_x = 1/(1 + e^{-(ms_x +c)})$ where $m, c$ are model hyper-parameters.

Why not use BCE Loss or methods proposed in [1] to achieve calibration?

We acknowledge that exploring the techniques in [1] could certainly be an interesting direction for future investigation to understand the effect of teacher models trained with different loss functions on student model performance, and we will add this as a future direction in our updated draft. Additionally, from our experience using BCELoss while training Bi-encoders leads to training instability. Hence we resorted to using pair-wise losses and posthoc calibration techniques

[1]: Scale Calibration of Deep Ranking Models, Le Yan et al, KDD 2022

[2]: Probabilistic Outputs for SVMs and Comparisons to Regularized Likelihood Methods, Platt 2000

Experimental settings and results

Model Size

Please note that the inference pipeline is unchanged with the addition of LEVER (both in terms of model capacity and inference time). LEVER only affects the training strategy by the addition of soft labels. So, the model size is the same for the base classifier and its LEVER counterpart. The below table shows the model size (in Millions of parameters) for the different models and datasets. Note the smaller model size in ELIAS for LF-AmazonTitles-1.3M is due to using 512 dimension classifiers, (as 768D led to OOM issues).

Significant gain in P@3/5 AOL in ELIAS

We apologize for the typo here, we accidentally entered values for ELIAS + LEVER on LFAT-1.3M here. The correct values were present in the appendix (Table 7), and now they have been added to the main paper in Table 1. as well. Here are the values for your reference

Significant gain in PSP metrics in WikiHierarchy in ELIAS

In ELIAS there is a bigger drop in head performance as compared to Renee when LEVER is applied. This therefore translates to a much larger increase in ELIAS+LEVER’s tail performance for this dataset, note that for ELIAS the performance improvement starts from the torso labels itself. This is illustrated in Figure 8 in the supplementary which compares decile wise plot ELIAS, ELIAS + LEVER and Renee, Renne+LEVER.

We thank the reviewer for their valuable feedback. Below, we provide clarifications.

The proposed method is just a combination of loss with hard and soft labels, which is a simple idea. Yes, while the idea is a combination of hard and soft labels, we would like to highlight that this simplicity allows the easy and effective combination of LEVER with any OvA-based XMC approach. However, unlike standard distillation where logits for all classes from teacher are used, in our case, for each label l, only the τ nearest points from the pool are selected and added as ground truth for distillation. This is how we have adapted distillation for our multi-label scenario. Table 22 and Figure 7 in supplementary show that increasing τ improves performance on tail, while it hurts head or torso label

If we have a model that provides good estimates then we don't need to train another one; the XMLC task is already solved!

Our intention for writing “if we have precise estimates of marginal relevance.. Reducing the variance to 0” was to convey that as the teacher gets more accurate, the variance term reduces to zero, and thus the upper bound on the true risk reduces, thereby improving generalization error of the classifier. However, a perfect teacher model is not available in practice. The NGAME model we use is a proxy for the perfect teacher and note that it is more accurate than the OvA classifier only on tail labels. Leveraging the strength of this Siamese model on the tail to reduce the label variance of OvA classifiers on the tail is what LEVER aims to achieve.

From the appendix, I understand that $\lambda$ (the variable that weights the hard and soft part of the loss) was used for all the experiments..

We think the reviewer has confused the $\lambda$ for ELIAS mentioned in Section D.2 of supplementary with the $\lambda$ for LEVER. We have renamed the $\lambda$ used in Section D.2 to $\lambda_{elias}$. The $\lambda$ used in LEVER is 0.5 (equal weight to both hard & soft labels).

I would like to see how different values of $\lambda$ impact the final performance and what the trade-off curve between head-labels and tail-labels performance looks like

In table below, we show the effect of varying $\lambda$ when LEVER is combined with Renee on 3 datasets: LF-AmazonTitles-131K, LF-Wikipedia-500K, & LF-AOL-270K. Note that $\lambda$=0.33 weighs the soft labels with double the weight as compared to hard labels. Our observations are: The equal weightage (0.5) gives the best performance. Increasing $\lambda$ weighs the hard labels more and this results in performance that gets closer to the base classifier, i.e., PSP worsens, and Precision remains flat or drops slightly. Additionally, decreasing $\lambda$ helps only up to a certain point, i.e., $\lambda$=0.5; this is because our teacher is not perfect we need to strike a balance between hard and soft labels.

This table has been added as Table 16 in the supplementary material. For decile-wise plots showing head-tail variation, please refer to Fig. 4, 5, 6 in the supplementary material

On the LF-AOL-270K dataset, LEVER combined with ELIAS yields extremely high improvement on standard precision@k..

Thanks for pointing this out. This is a typo; the number from the LF-AmazonTitles-1.3M ELIAS+LEVER row was mistakenly copied to the ELIAS+LEVER row in LF-AOL-270K. The supplement however has the correct values for LF-AOL-270K without this typo. We fixed this in the main table of updated draft. Here are the values for your reference

---

We hope this rebuttal clarifies your concerns. We would be happy to discuss any further questions, and would appreciate an appropriate increase in the score if the reviewer’s concerns are addressed to facilitate the acceptance of the paper.

We thank the reviewer for their valuable feedback. Below, we provide clarifications:

Concerns around novelty: The paper claims that the variance issue for tail labels has not been explored so far. However, this seems not quite true, as a similar concern has been raised in the earlier work by Babbar and Scholkopf 2019

[1] discusses the issue of variance from the point of view of a lack of commonality between features of training instances and those between training and test instances. However, the definition of variance we use is different, and we highlight it with an example below:

Consider a dataset where data points are user queries and labels are advertisements shown in response to user queries. Now, if we consider a label with the text “insurance,” there could be a wide variety of user queries (or documents), such as “hospital bills” and “car repair,” that are relevant to this label. This lack of commonality between documents associated with a single label has been discussed in [1].

In recommendation tasks, dataset rely on click signals to build the ground truth, i.e., if the user clicks on an advertisement for a particular query, then the query-advertisement pair is considered relevant (1 in ground truth). However, feedback in the form of user clicks is subject to variability as a user’s interests can fluctuate over time; for example, for the query “hospital bills,” there may or may not be a click on the “insurance” ad. Approximating the relevance using click signals that are binary leads to inaccuracies in the dataset, which we quantify using the term label variance.

In regards to the novelty of our work, we would like to highlight a few points:

1. We propose the concept of label variance, which, to the best of our knowledge, has not been explored in the domain of extreme classification.

2. While Menon et al. discuss the teacher-student setup, their teacher is better than the student overall (as indicated by superior Precision values in Table 1). While such a strong teacher might not always be available, we demonstrate the potential of improving the performance of OvA classifiers by proposing a Siamese teacher that excels only on tail labels. Note that the Siamese teacher is inferior in precision compared to the OvA student model. To the best of our knowledge, we are the first to propose distilling from a teacher that excels only on a subset of the dataset population (tail labels) for XMC tasks.

3. While the idea is simple, we believe the method's simplicity is a strength as it allows for an easy and effective combination with state-of-the-art one-versus-all methods, as highlighted in our results.

[1]: “Data scarcity, robustness, and extreme multi-label classification” (Babbar and Scholkopf 2019)

Inconsistent notation in Section 3.1, and better presentation of improving presentation of Theorem 1.

Thanks for highlighting this. We will work on addressing the issues in our final draft.

Citing related works

Justification for the use of calibrated losses

In Section 3.3, we discuss the need to use calibrated losses. In summary, Siamese networks trained using triplet loss have shown to do well on tail labels. However, the scores output by these models are not well calibrated, i.e., they do not have a probabilistic interpretation). For training OvA classifiers with soft targets, we need calibrated scores, and Theorem 2 in our paper shows how we can obtain calibrated scores from the Siamese teacher. Note that this is in line with the posthoc calibration strategy mentioned in [3], where a model is learned. Then a parametrized Sigmoid function is fit to learn the relevance probabilities, i.e., given an uncalibrated score $s_x$ we get $p_x$ by fitting the sigmoid function $p_x = 1/(1 + e^{-(ms_x +c)})$ where $m, c$ are model hyper-parameters. We will clarify the definition of calibration used in the current context and add a citation to [2].

Meaning of calibration used at the end of Pg1?

At the end of Pg 1. We discuss the strategies that rebalance the loss functions to tackle the problem of extreme class imbalance in XMC datasets. We will clarify this in the final version of the paper.

Citing work of Schultheis et al. 2022 in the correct context and missing citation of DiSMEC

In our revised draft, we have cited the work of Schultheis et al. 2022 in Section 2.2 while discussing “Bias due to Missing Lables,” and DiSMEC has now been cited in the introduction while discussing about OvA classifiers in the revised draft.

We hope this rebuttal clarifies the concerns you had about the paper. We would be happy to discuss any further questions about the work. We would really appreciate an increase in the score if the reviewer’s concerns are adequately addressed to facilitate the acceptance of the paper.

[2]: Scale Calibration of Deep Ranking Models, Le Yan et al, KDD 2022 [3] Probabilistic Outputs for SVMs and Comparisons to Regularized Likelihood Methods, Platt 2000