The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators

Response to Reviewer b4or

We are grateful for the review and the positive assessment. Thank you for acknowledging Alchemist is novel, interesting, and our paper is well-written. We address your questions below.

• On Performance Degradation.
  • In the majority of cases (six out of eight datasets), Alchemist performs as well as or even better than model-based annotation, in addition to being orders of magnitude cheaper, more extensible, and easier to inspect. In fact, our results suggest that users should prefer Alchemist over model-based annotation.
  • In cases where a user seeks to improve Alchemist's base performance, we can simply add more generated programs---at extremely low cost. In Table 1 in our paper, we report results with 10 generated programs. Increasing the number of generated programs enhances performance, as we showed in Figure 5, where generating 20 labeling programs for Yelp and IMDb datasets results in better labeling performance compared to zero-shot prompting.
• On Accuracy Performance in Table 1 and Table 3.
  • Results shown in Table 1 are for a model trained on Alchemist-created labels (we report performance on the testing dataset), while in Table 3, we present the label model's performance (i.e., just using the ensemble of programs that Alchemist generates) on the testing dataset. These two approaches correspond to two stages in the weak supervision process: first, create a label model that just uses the labeling functions (programs, in our case), and second, obtain labels from the label model and train an end model. Using just the label model and training an end model have both been extensively studied in the weak supervision literature. It is known that training an end model on weakly-labeled data enables generalizing beyond the label model in many cases. The reason for this is based on the observation that the label model may not be capable of capturing all labeling patterns (i.e., all possible features). It may instead only use certain simple ones. Training an end model permits learning many patterns, both simple and complex, which enables better generalization in many scenarios. Our results are consistent with these notions.
• On Labeling the Test Set.
  • We described the difference between the label model and the end model above. We organized Alchemist's results in Table 5 in our attachment. We show the testing performance using the label model and the trained end model for eight datasets. In most of the cases, the end model is preferable to the label model.
• On Techniques to Evaluate Generated Programs.
  • There are many ways to evaluate the quality of generated programs in advance. First, expert users can quickly determine whether key problem properties are being used by looking at the code. Besides human inspection, Alchemist includes several automated measurement tools to diagnose generated programs. We analyze program outputs and compute their coverage, polarity, conflict, and overlap (see [1] for definitions). For instance, coverage measures the fraction of data points with at least one label. If coverage is below a certain bar (e.g., 10%), we discard the program and ask users to generate a new one.
  • Moreover, if a validation dataset is available, Alchemist can run diagnostics to empirically compare accuracy with ground truth, offering more insight into the program's reliability. Notably, these tools are not typically accessible with model-based annotation methods.
• On Data Characteristics.
  • In general, we have found that Alchemist works well when the labeling logic involves a mixture of logical formulas over a set of salient primitives. This captures most cases of interest. Formalizing a definition of what data or task characteristics are relevant is a very interesting problem for future work; we foresee studying, for example, the expressivity of programs and understanding if they can meet most ML tasks. Kolmogorov complexity or similar notions will likely be helpful here.
• On Framework Limitation.
  • This limitation is common across many annotation approaches. We often only know the results after completing the entire labeling process, which is also true for model-based annotation methods. However, Alchemist significantly reduces expenses, requiring a lower order of magnitude and enabling more efficient reruns of the labeling process. In other words, Alchemist offers a much more tractable approach to iteration compared to model-based annotation.

[1] https://snorkel.readthedocs.io/en/v0.9.3/packages/_autosummary/labeling/snorkel.labeling.LFAnalysis.html

Response to Reviewer d5ZD

Thank you for recognizing Alchemist’s flexibility, extensibility, and for your kind words about our paper! We address your questions below and include experimental results on more complex tasks and program stability.

• On More Complex Tasks.
  • Alchemist is not limited to threshold-based simple tasks. It is capable of capturing more complex labeling logic, as we have observed in some of the generated programs. To further demonstrate this capability, we included two more challenging datasets: ECG heartbeat classification and Census income classification. Results are shown in Table 2 in our attachment and demonstrate that Alchemist can handle more complex modalities and perform well. Moreover, we compared Alchemist with human-crafted labeling functions from WRENCH. Alchemist uses fewer labeling functions (programs) and reaches higher labeling performance.
  • Additionally, since submission, we have been working on an additional, highly timely, application of this work to LLM-as-a-judge. This is a crucial area since users and companies spend vast sums of money for evaluating model outputs using automated LLM-as-a-judge techniques. We have collaborated with a leading startup in the industry to use our technique in production environments to reduce costs. Our preliminary results in this area suggest that Alchemist can capture sufficiently complex logic to identify poor-quality responses from language model pipelines.
• On the Stability of Generated Programs.
  • This is a great question. We conducted an experiment by varying the temperature in our query APIs and running Alchemist on four different datasets. We trained the end model five times with different random seeds and computed the average performance and the variance. Results are shown in Table 4 in our attachment. We observe consistent labeling performance across different temperatures (0.0, 0.5, and 1.0), demonstrating Alchemist’s stability. Additionally, the stability of generated programs highlights the significance of including aggregation models to handle noisy and diverse outputs, resolve conflicts, and produce final labels.
• On Improving Literature Review.
  • Thank you for sharing these papers with us. We have included them in our related work section. Key differences: These papers prompt LLMs for labels per-sample, requiring a large number of API calls (scaling with the size of the dataset) and making it hard to inspect mistakes and revise labeling logic, unlike Alchemist.

Response to Reviewer n36E

Thank you for recognizing our paper as well-written with well-designed experiments. We appreciate your acknowledgment of our simple, effective idea and the significance of including diverse modalities. We appreciate your thoughtful review!

• On Comparisons with Related Work.
  • We compared Alchemist with related works in Table 1 in our attachment. Alchemist uses fewer programs while achieving comparable labeling accuracy. Key differences include:
    • Motivation: Both ScriptoriumWS and DataSculpt address weaknesses in weak supervision, whereas Alchemist focuses on the downsides of model-based annotation.
    • Approach: In ScriptoriumWS, supplementary information is manually crafted into prompts to generate code, while Alchemist uses LLMs to distill self-knowledge, enhancing labeling performance but using less human effort. DataSculpt employs hundreds of programs for obtaining high-quality labels, but Alchemist achieves comparable accuracy with just 10 programs.
    • Complex Modalities: Alchemist handles complex modalities beyond text, while ScriptoriumWS and DataSculpt do not.
  • PromptWS significantly differs from Alchemist. PromptWS prompts LLMs multiple times for each sample, resulting in more API calls (even compared to zero-shot prompting). While this method can improve performance, it incurs extremely high labeling costs, is harder to audit, and lacks extensibility. Alchemist, in contrast, addresses these issues.
• On Suggested References.
  • Thank you for sharing this paper. We have updated our related work and included it. While this work shares a similar idea with Alchemist, Alchemist goes beyond keyword extraction by capturing more complex and diverse labeling logic, and organizing them into executable programs (see Figure 1 in the paper). Moreover, extending to non-text modalities like image classification is challenging, but Alchemist is capable of this.
• On Reproducibility.
  • We attached our code in the supplementary material of our submission. We will release our code and a full set of outputs, including generated programs, after paper notification.
• On the Data-Driven Feedback Loop.
  • This is a good question, and it is a crucial step. Alchemist includes several tools for diagnosing generated programs. First, we analyze program outputs to compute coverage, polarity, conflict, and overlap (see [1] for definitions). For example, if coverage (the fraction of data points with at least one label) is below 10%, we discard the program and ask users to generate a new one. Second, if a validation dataset is provided, Alchemist computes empirical accuracy by comparing outputs with ground truth. Such a data-driven feedback loop ensures tractable program generation. We have included this step's description in the updated draft.
• On Richer Modalities.
  • Thank you for the suggestion! We included two additional modalities in Table 2: time-series (ECG heartbeat classification [2]) and tabular (Census income classification [3]). For ECG heartbeat classification, we generated 10 labeling programs from GPT-4o. For the Census income dataset, we generated program codes for each attribute (e.g., gender, education, age, race). We used Snorkel as our label model. The results demonstrate Alchemist's capability to handle more complex modalities and produce satisfactory performance. Moreover, we compared Alchemist with human-crafted labeling functions from WRENCH. Alchemist uses fewer labeling functions (programs) and reaches higher labeling performance. In general, Alchemist will work well with any of these modalities as long as we have access to any cheap local feature extractor. This includes medical imaging tasks: [4] showed manually-crafted simple labeling functions were able to identify heart problems in MRI sequences based on very simple primitives, which could act as the feature extractors for Alchemist.

[1] https://snorkel.readthedocs.io/en/v0.9.3/packages/_autosummary/labeling/snorkel.labeling.LFAnalysis.html
[2] https://physionet.org/content/mitdb/1.0.0/
[3] https://archive.ics.uci.edu/dataset/20/census+income
[4] Fries, J.A., Varma, P., Chen, V.S. et al. Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences. Nat Commun 10, 3111 (2019).

Dear reviewer,

Thank you for your valuable feedback! Your suggestions have improved our paper. Appreciate it. We answer your questions below.

1. Thank you for your suggestion! We agree and have added additional descriptions to the paper addressing the output-driven feedback loop. One way to handle the challenge of making this work in non-text modality settings is to develop a set of simple unit tests that can be used to drive the feedback loop.

2. The difference comes down to what the goals of these techniques are. ScriptoriumWS and DataSculpt focus on automating weak source (labeling function) creation, addressing the limitations of human-crafted labeling functions, which require subject matter experts to implement by hand. The ideal outcome of these systems is to speed up the process of performing weak supervision by reducing the complexity of writing correct code. In contrast, Alchemist addresses the downsides of model-based annotation, such as high cost, lack of extensibility, and difficulty in auditing. Its goal is to efficiently distill model capabilities. While the mechanisms currently used to accomplish these goals have similarities, they are motivated by different challenges.

3. Indeed, ScriptoriumWS does not operate on non-text modalities at all; that technique is to directly prompt a language model to generate programs/labeling functions that perform a text task. The most straightforward way to extend the ScriptoriumWS methodology is to prompt a multimodal model like GPT4o to generate programs over the target modality (e.g., images). There are two downsides we have identified to this approach to extending ScriptoriumWS:

   • It requires access to a powerful multimodal model, which may not exist for many modalities of interest,
   • Even if such multimodal models exist, they may struggle with spurious correlations.

While we did not directly evaluate ScriptoriumWS with GPT4o, we did extend Alchemist itself in this fashion as a baseline. In Table 2 in our paper, we reported our findings, which suggest that the spurious correlation issue is indeed a problem. In contrast, Alchemist’s approach, based on obtaining primitives and using a cheap feature extractor, reduces the impact of these spurious correlations. It additionally also mitigates the other challenge, as such feature extractors are much easier to obtain compared to a powerful multimodal model.

4. In this setting, we did not use the two-stage extension method to generate programs. Instead, we included supplementary information, such as dataset and prediction class definitions, directly in the prompts. Following the basic recipe for our work, we generate 10 programs and use Snorkel as the aggregation method to produce predictions. We show two generated programs below as examples. They use peak count as their labeling logic. The first program achieved 69.5% accuracy, while another reached 81.5%.

def label_by_fft_peak(time_series):
    """ Label based on the frequency domain characteristics """
    fft_result = np.fft.fft(time_series)
    fft_magnitude = np.abs(fft_result)
    peak_freq = np.argmax(fft_magnitude[1:]) + 1  # Ignoring the DC component
    if peak_freq < 15:
        return 0  # 'N'
    elif peak_freq < 30:
        return 1  # 'S'
    elif peak_freq < 45:
        return 2  # 'V'
    elif peak_freq < 60:
        return 3  # 'F'
    else:
        return 4  # 'Q'

def label_by_peak_count(time_series):
    """ Label based on the number of peaks in the signal """
    from scipy.signal import find_peaks
    peaks, _ = find_peaks(time_series, height=0)
    peak_count = len(peaks)
    if peak_count > 7:
        return 0  # 'N'
    elif peak_count > 5:
        return 1  # 'S'
    elif peak_count > 3:
        return 2  # 'V'
    elif peak_count > 1:
        return 3  # 'F'
    else:
        return 4  # 'Q'

Response to Reviewer uwkF

We are grateful for your review and for describing our paper as significant, well-written, and of high quality. We address your questions below and provide additional experimental results incorporating Roboshot into Alchemist!

• On Grounding.
  • Thank you for sharing this paper with us. We agree it relates to Alchemist, and we have included it in our updated related work. Key differences include: unlike Alchemist, which automatically generates programs, this work requires domain expertise and more human effort to craft heuristics and search for proper keywords.
  • In Sec. 4.4 in our paper, we include an experiment comparing Alchemist to human-crafted labeling functions. For example, in the SMS dataset, WRENCH requires 73 manually crafted labeling functions to obtain high-quality labels, whereas Alchemist only needs 10 generated programs to achieve comparable performance and higher coverage. This significant reduction highlights Alchemist's capability to assist humans in generating label sources, making it more accessible to users without extensive domain expertise.
• On Two Similar Works.
  • Thank you for pointing out these two papers. The first paper prompts LLMs for spurious and correct correlation features, then calibrates image embeddings by projecting them to reject or accept concepts. The second paper asks LLMs for confusing visual objects to perform zero-shot one-class classification with CLIP. We have included them in the draft.
• On Incorporating Roboshot into Alchemist.
  • This is an interesting idea. We implemented this suggestion by incorporating Roboshot into Alchemist. We first used GPT-4o to identify spurious correlations for classifying waterbirds and landbirds, then projected image embeddings onto these concept embeddings to reject spurious correlations through subtraction. We computed cosine similarity using the calibrated embeddings to obtain score sets, which were then fed into Alchemist’s generated programs. The spurious correlations identified by GPT-4o were {'water background,' 'land background', 'aquatic plants,' 'trees and bushes'}. Results are displayed in Table 3 in the attachment. We observe that integrating Roboshot with Alchemist using GPT-4o enhanced robustness to spurious correlations by improving accuracies.
• On Framework Limitations.
  • Our motivation is to address the drawbacks of pretrained model-based annotation, which is expensive, lacks extensibility, and makes results hard to inspect. We believe Alchemist serves as a better labeling tool by reducing costs and providing human labelers with more power to inspect and refine labeling results. In addition, generated programs can serve as templates for human labelers to rewrite, extend, or customize labeling logic. These benefits make the labeling process easier and more efficient for humans.