Sanitizing LLMs: Retrospective Learning for Self-Correction of Inconsistent Samples via User Preferences

Comment 3: In addition, the motivation of the work is not super clear to me either.

Dear Reviewer, Thank you for your question regarding the motivation of our work. Let me clarify this further. In unsupervised learning tasks that rely on user preferences to align data annotations with expectations—where the competency of the teacher model (LLMs) is uncertain and no external knowledge is available—the key challenge lies in evaluating the annotations generated by LLMs and enabling mechanisms for self-correction. To address this, we propose a novel approach termed Retrospective Learning (ReL), a self-supervised framework designed to facilitate both self-correction and self-assessment of annotations produced by LLMs.

Our methodology leverages a student model to collaborate with a teacher model of uncertain competency. By introducing the Consistent and Inconsistent (CAI) score and (CAI) identification, we systematically quantify and identify consistent and inconsistent samples. This iterative refinement process enhances the performance of both the student and teacher models. To demonstrate the efficacy of our approach, we conducted experiments using the Meta-8B Instruct lightweight LLM as a low-competency “noisy teacher.” This setup, combined with the student model, underscores the robustness of our framework and highlights the pivotal role of the student model in managing scenarios involving noisy teachers. In summary, given the challenges of unsupervised learning tasks—where the teacher’s competency is unknown and no external knowledge is available—we aim to answer: How can we evaluate and enable self-correction for an unsupervised dataset? Our proposed Retrospective Learning strategy addresses this by enabling both self-correction and self-evaluation of LLM-generated annotations. Through collaboration between the student and teacher models, the CAI score helps identify consistent and inconsistent samples, ultimately improving the performance of both models.

Thank you again for your question, and we hope this explanation clarifies the motivation of our work.

Table: ChatGPT-4o Mini (Closed-source LLMs)

Dear Reviewer,

Thank you for your insightful feedback regarding the reliability of the CAI score and its dependency on the teacher and student models. To address your concerns, we have expanded our experimental study to include three additional NLP datasets, bringing the total to eight: Bank77, CLINC (Intent), MTOP (Intent), Massive (Intent), StackExchange, Reddit (Topic), Few Rel Nat (Type), and Massive Intent (Intent). We have also incorporated an open-source lightweight LLM (Meta-Llama 8B Instruct) alongside two closed-source LLMs (ChatGPT-3.5 Turbo and ChatGPT-4 Mini), increasing the total number of evaluated models to three. Our study evaluates the reliability of the CAI score and the effectiveness of our proposed Retrospective Learning (ReL) framework across these diverse LLMs and datasets. The updated results, presented in the main paper (Section 4.3, Experimental Results) and appendix (Sections A.2 and A.3), demonstrate the robustness of the CAI score in identifying LLM performance in unsupervised tasks with user preferences.

Results Summary

1. Pearson Correlation Analysis for ChatGPT-3.5 Turbo and ChatGPT-4 Mini (3 Additional Datasets):

Key Observations

• Effectiveness of CAI Score: The CAI score consistently demonstrates a strong positive correlation with accuracy across all tested models and datasets, validating its reliability in evaluating LLM performance.
• Performance Discrepancy Between LLMs: Meta-8B Instruct consistently achieves lower CAI scores compared to ChatGPT-3.5 and ChatGPT-4 Mini, aligning with its lower average accuracy across datasets. This highlights the CAI score's ability to reflect model performance and reliability effectively.

Our CAI score has proven effective across both closed-source LLMs (ChatGPT-3.5 Turbo and ChatGPT-4 Mini) and open-source lightweight LLMs (Meta-Llama 8B Instruct), as evidenced by statistically significant results from the Pearson correlation analysis. We have updated the main paper to include these new results (see Section 4.3, Experimental Results), along with detailed statistical analyses in the appendix (see Section A.2.1, Statistical Inference). We hope these expanded experiments and additional explanations address your concerns. Thank you again for your valuable feedback, which has significantly improved the scope and depth of our work.

Dear Reviewer

We have added a more detailed explanation regarding the clustering operation in the following:

Clustering Operations in Retrospective Learning

The clustering operation is conducted using a semantic similarity score equation and majority voting based on a top-nearest embedding scheme. This process is crucial in our retrospective learning framework, aligning annotations with user-defined preferences. By assigning annotations in this manner, the Consistent Annotation Identification (CAI) method facilitates self-correction of identified inconsistent samples through the Divide-and-Conquer Self-Correction (DCSC) process—an iterative self-correction mechanism. Our approach supports user-defined preferences and is applicable to a wide range of large NLP datasets, making it particularly impactful for unsupervised tasks involving user preferences.

Steps of the Clustering Operation:

1. Extracting Embeddings from the Student Model

   Semantic features of text inputs are obtained using a pretrained student model (e.g., MiniLM) to generate dense vector representations. For a given sample $x_i$, the embedding function $S(x_i)$ computes an embedding $e_i$, capturing the sample's semantic features.

2. User-Preference Sample Clustering

   A small annotated set of user-preference samples $H = \{x_1, \dots, x_s\}$ is provided. These samples are partitioned into $k$ clusters, denoted as $C_1, C_2, \dots, C_k$, based on predefined labels $y_j \in \mathcal{Y}$, which reflect the user’s annotation preferences. The clusters satisfy the conditions:

   $C_i \cap C_j = \emptyset, \quad \forall i \neq j, \quad \text{and} \quad \bigcup_{i=1}^k C_i = H$.

3. Annotation Assignment for Unlabeled Data Using the Student Model

   For each unlabeled sample $x_i \in D_u$, its embedding $e_i = S(x_i)$ is computed. The semantic similarity of $e_i$ to each cluster $C_j$ is calculated using the Average Similarity (AS):

   $AS(e_i, C_j) = \frac{1}{k} \sum_{e \in \text{Top-}k(C_j, e_i)} \frac{e_i \cdot e}{\|e_i\| \|e\|}$,

   where $e$ represents the embeddings in $C_j$, and $\text{Top-}k(C_j, e_i)$ is the subset of samples in $C_j$ with the top $k$ cosine similarity scores to $e_i$. The cluster $C_{j^*}$ with the highest $AS(e_i, C_j)$ is selected, and its label $\bar{y}_{j^*}$ is assigned to $x_i$. The resulting student-annotated dataset is denoted as $D_s = \{(x_i, \bar{y}_i)\}$.

4. Annotation Assignment for Unlabeled Data Using the Teacher Model (LLMs)

   Using $D_s$, annotations are refined with a teacher model (LLMs) via two prompting strategies:

   • Zero-shot prompting: No annotations from the student model are included.
   • Single-shot prompting: Student annotations are incorporated.

5. Identification of Consistent and Inconsistent Samples

   In the absence of ground truth annotations, we introduce the CAI score, a metric for evaluating annotation quality in unsupervised tasks. This score identifies consistent and inconsistent samples by comparing the annotations from the student and teacher models. Further details on the CAI score and its calculation are provided in Section 3.2 of the main paper.

6. Majority Voting via Top-Nearest Embedding Scheme (MV-VTES) and Divide-and-Conquer Self-Correction (DCSC)

   To correct corrupted annotations in inconsistent samples, we propose MV-VTES and the DCSC framework.

   • MV-VTES: Uses consistent samples and user-preference samples to iteratively refine annotations for inconsistent samples based on majority voting within the top-nearest embeddings.
   • DCSC: A divide-and-conquer mechanism that iteratively corrects annotations for identified inconsistent samples.

   Consistent samples, having significantly higher accuracy, serve as a reliable reference for annotation refinement. For detailed algorithms, refer to Section 3.3 of the main paper.

Table: Meta-Llama 3-8B Instruct (Open-Source Light-Weight LLMs)

Annotation Accuracy comparison in percentages with standard deviations across different datasets for the Student Model, LLMs without annotations from the Student Model, and LLMs with annotations from the Student Model. The highest accuracy for each dataset is highlighted.

Comment.2: In fact, my primary question throughout the entire paper is why a student model is necessary at all; this aspect does not seem to be addressed in the introduction. The motivation seems to lack clarity and justification.

Dear Reviewer 4bru,

Thank you for your feedback. One of the key motivations for our approach, including the use of a student model, is to enhance efficiency. This efficiency is evident in two significant aspects:

1. Computational Efficiency: Our method requires access to the teacher model only twice per dataset, with minimal or no reliance on demonstrations for prompting. This substantially reduces computational overhead.

2. Cost-Effectiveness: For closed-source models with API service fees, our approach offers a cost-efficient solution. By utilizing the student model alongside our proposed clustering operation and a limited number of teacher model predictions, our method achieves superior performance compared to both models individually. Importantly, it does so at a lower cost, particularly when compared to methods that rely on iterative self-correction and feedback loops.

We hope this explanation provides additional clarity and highlights the practical benefits of our approach.