Confidence Difference Reflects Various Supervised Signals in Confidence-Difference Classification

Q1. If $c>0.5$, the two examples belong to different classes. This is not true.

Thank you for the insightful correction. The statement "$c>0.5$, the two examples belong to different classes" is indeed not precise enough. A more accurate expression would be "$c > 0.5$, the two examples belong to different classes at a high probability". This is also consistent with the data partitioning strategy adopted in our paper, which separates the dataset into a subset $D^S$ with relatively precise predictive information (i.e., $c > 0.5$) and a subset $D^C$ with comparatively imprecise predictive information (i.e., $c \le 0.5$). Moreover, we agree with your point that, there is no direct correspondence between the labels and the posterior probability during training. Our goal is to enable the model to learn and establish this relationship through training.

We will revise this statement to be more rigorous in the next version.

 

Q2. I am not sure whether equation (7) is biased from the original classification risk in equation (1). This is because the marginal distribution may change after data partitioning. If so, will this affect the theoretical analysis in Theorem 3.1, since the minimizers of the two risks are not the same?

Thank you for your comments. We agree that Eq.7 constitutes a biased estimator of the original classification risk in Eq.1, due to the change in the marginal distribution induced by the subset partitioning strategy. However, this bias does not affect the theoretical analysis in Theorem 3.1.

In Theorem 3.1, we explicitly model the risks over the two subsets separately, and analyze their contributions through the Rademacher complexities $\mathfrak{R}\_{n_1}(\mathcal{G})$ and $\mathfrak{R}\_{n_2}(\mathcal{G})$, respectively. The first and second terms of the error bound directly involve these complexities. So as $n_1, n_2 \to \infty$, both $\mathfrak{R}\_{n_1}(\mathcal{G})$ and $\mathfrak{R}\_{n_2}(\mathcal{G})$ tend to zero. Additionally, the third term, which depends on $\sqrt{n}/n_1$ and $\sqrt{n}/n_2$, also diminishes as the sample sizes increase. Consequently, as long as $n_1, n_2 \to \infty$, the overall estimation error still converges to the minimum of the original classification risk $R(g^*)$.

 

Q3. Line 183 should read instead $g$ of $\mathcal{G}$. The condition in line 184 is also incorrect.

Thanks for your correction. We will revise them in the next version.

Q1. Which specific loss functions (e.g. logistic) are included in Eq.5?

Thank you for your comments. This function class includes several commonly used loss functions, such as those derived from Generalized Linear Models (GLMs), including the mean squared error (MSE) for linear regression, the logistic loss for logistic regression, the Poisson loss for Poisson regression, the exponential loss for exponential regression, and the cross-entropy loss for neural networks (Zhang et al., 2021).

Zhang, L., Deng, Z., Kawaguchi, K., Ghorbani, A., and Zou, J. How does mixup help with robustness and generalization? In International Conference on Learning Representations, 2021.

 

Q2. Could you provide a clearer explanation of the x-axis in Fig.1?

Thank you for your suggestions. The x-axis values multiplied by $\pi$ represent the proportion of pairwise instances $(\mathbf{x}, \mathbf{x}')$ with confidence differences $c(\mathbf{x}, \mathbf{x}') \in [-1, -0.5) \cup (0.5, 1]$ relative to all pairwise instances. Thus, the x-axis values effectively serve as a scaling factor used for computing the proportion.

 

Q3. What is the purpose of the design in Section 3.4? Why might negative empirical risk lead to severe overfitting? And how does the risk correction function address this issue?

Many thanks for your comment. The purpose of Section 3.4 is to address the overfitting problem when using flexible models due to negative empirical risk.

Risk is typically non-negative, reflecting the deviation between model predictions and ground-truth values. The objective of optimization is to minimize risk; if risk could be negative, the model could be inclined to find an optimization direction that continually reduces risk on the training data, leading to overfitting by learning noise and performing poorly on test data. Additionally, (Lu et al., 2020) highlights that negative empirical risk may be a potential cause of overfitting and experimentally demonstrates a strong co-occurrence of negative risk and overfitting across various models and datasets.

The risk correction function enforces the non-negativity of the risk by using $|\cdot |$ or $max\\{0,\cdot\\}$.

Lu, N., Zhang, T., Niu, G., and Sugiyama, M. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. In International Conference on Artificial Intelligence and Statistics, pp. 1115–1125. PMLR, 2020.

 

Q4. Would consistency regularization term lead to more instance pairs with smaller confidence differences, thereby increasing the presence of noisy supervised signals?

Thank you for your comments. The core objective of consistency regularization is to enforce consistency in the classifier's predictions for pairwise instances with small confidence differences by constraining the model output. It is important to clarify that our optimization target is the classifier's output $g(\cdot)$, not the confidence difference $c$. In our setting, $c$ serves as an attribute used for training, functioning as a form of weak supervision. Our goal is not to optimize $c$, but to use it to guide the classifier toward the desired outputs. Therefore, the concern about "causing more pairs to have smaller confidence differences" does not apply here.

Q1. The motivation of this paper is not strong enough, and what real-world scenerios can motivates this problem? Thank you for your suggestions.

About motivation. The motivation for our method arises from the observation that small confidence differences may lead to imprecise guidance within $R_{CD}$, particularly when the confidence difference equals zero, resulting in a complete lack of predictive guidance. Our experiments in Figure 1 further demonstrate that pairwise instances with larger confidence differences dominate the contribution to $R_{CD}$, while those with smaller confidence differences contribute minimally. To address this issue, we propose a consistency regularization term that encourages $\mathbf{x}$ and $\mathbf{x'}$ to produce more similar outputs in the model when $|c(\mathbf{x}, \mathbf{x'})|$ is small.

About real-world scenerios.

1. Rehabilitation assessment. The assessment of whether a patient meets rehabilitation criteria presents significant challenges. Individual differences in recovery make the evaluation process inherently subjective. In addition, assessments become particularly uncertain when a patient is close to the threshold of rehabilitation. In contrast, clinicians can more reliably generate approximate confidence labels using objective data, such as scores from functional assessment scales (e.g., FIM or Barthel Index), questionnaire responses, and motor performance metrics. Moreover, by considering the continuity of the rehabilitation process and individual differences, changes in a patient's condition over time can be used to construct confidence difference that reflect recovery trends, leading to more robust rehabilitation assessment.

2. Click-through rate prediction. In recommender systems, predicting whether a user will click on a given item is a central task. However, due to the sparsity of click data and the problem of class imbalance, it is often difficult to assign accurate pointwise label to each user-item pair. In contrast, collecting pairwise preference information, which refers to the relative preference between candidate items for a given user, is a more feasible and effective alternative. In practical applications, approximate confidence can be obtained using auxiliary probabilistic classifiers, such as models predicting click probabilities. Moreover, in many recommendation scenarios such as news, short video, and movie recommendations, real-valued feedback like watch ratio or user ratings is often available, which can be leveraged to construct more informative confidence difference.

In addition, it also holds practical value in various domains such as obstacle detection in autonomous driving, driving behavior analysis, and financial risk assessment.

 

Q2. It seems that this problem is only applicable for binary classification, which further limits its application in real-world scenerios.

Thank you for your insightful comments. The method proposed in this paper is indeed developed within the framework of binary classification. Nevertheless, we respectfully believe that this setting does not substantially limit its applicability to real-world scenarios. First, a wide range of real-world applications naturally take the form of binary classification problems, where our method can be directly implemented. Typical examples include medical diagnosis, rehabilitation assessment, and financial risk management. Furthermore, the proposed method is conceptually general and can be naturally extended to multi-class classification tasks, which we consider a promising direction for future research. Formally，let $c_i \in \mathbb{R}^l$ be the confidence difference between pairwise unlabeled data $(\mathbf{x}_i,\mathbf{x}'_i)$ drawn from an independent identically distribution probability density $p(\mathbf{x},\mathbf{x}')=p(\mathbf{x})p(\mathbf{x}')$:

$\mathbf{c}_i = [c_i^{(1)}, c_i^{(2)},\dots ,c_i^{(l)}],\\:\\:c_i^{(k)}=c^{(k)}(\mathbf{x}_i,\mathbf{x}'_i)=p(y'_i=k|\mathbf{x}'_i)-p(y_i=k|\mathbf{x}_i)$

where $l$ denotes the number of classes. Accordingly, the consistency regularization term over $D^{C}$ in the expected risk can be modified as:

\alpha \mathbb{E}\_{p\_{{\small \mathcal{D}^{C}} }(\mathbf{x},\mathbf{x}')} [ \bigl(\frac{1}{ \log \left(\left| \mathbf{c(\mathbf{x},\mathbf{x}')}\right|_1 + \varepsilon \right) } \bigr) \cdot \left \\| g(\mathbf{x})-g(\mathbf{x}') \right \\|_2 ]

In summary, this work proposes a general framework for addressing noisy supervision signals in confidence difference classification. The proposed method is not limited to binary classification tasks and also demonstrates practical applicability in real-world scenarios.