SCALE: Augmenting Content Analysis via LLM Agents and AI-Human Collaboration

1. In the introduction, the authors mention that one of the drawbacks of using humans experts is the time and labor cost. The analysis of the proposed framework would benefit significantly if there is any analysis in terms of time/cost spent by humans for annotation vs LLMs.
2. I think Section 5.3 SUPERIOR PERFORMANCE OF SCALE should emphasize the overall quality of the whole framework instead of the single coding accuracy. The classification task is relatively trivial for LLMs.
3. Human evaluation (or detailed results) might be needed to assess the overall quality of using LLM agents to simulate content analysis beyond Codebook Update Phase.

The central Figure (Figure 2) is not that easy to understand. It should be self-explanatory by reading the caption. It would be very helpful to know which dataset is taken as an example here. Unfortunately, the paper does not always read fluently. Sometimes articles are missing and there are some grammatical errors. Some technical details are missing, such as the implementation of the chain of thought and tree of thought baselines (or at least the references are missing - see below). Also the formula for the used accuracy measure should be written out (in my opinion). The human intervention experiment is not really explained well. How much did the humans intervene? Is there a certain number of rounds? Is it the same setup as in the previous experiments? Overall the idea of the framework and the process of inter-agent discussion for automated content analysis is good, but some important details are missing. It is also not clear from the paper how much manual effort is required to apply the whole framework. What are the necessary steps (e.g. developing personas, a first version for a codebook..)? As the authors note at the end the inter-agent discussion introduces significant computational overhead. This leaves the question how practical the framework is.

Missing references:

• Chain of thought prompting as introduced by Wei et al (2022): Wei, Jason, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 2022, 35. Jg., S. 24824-24837.
• Tree of thougt prompting: Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., & Narasimhan, K. (2024). Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36.
• Self Consitency: Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., ... & Zhou, D. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.

Small errors:

• Line 088 should probably have a period at the end of "Human Intervention" to be consistent with the other items.
• line 190 is missing a period before (c)
• line 208: it should be N personas P, which ..., are derived from.. s
• The abbreviation NES is not introduced in the text
• line 241: "a K-round discussion" instead of "an K-round.."
• Line 320: It would be very nice if the Hamming loss was explicitly written out here in the formula.
• Line 461: lLM > LLM

Despite focusing on an interesting and promising idea, this manuscript presents different criticalities, as follows:

• There is much emphasis on the capability of SCALE to incorporate domain knowledge of social science, yet this process seems limited to one (of many possible) prompting strategies, undermining the robustness and technical depth of the proposed framework.
• The experimental setup is not appropriate, as there is no comparison with baseline models (e.g., ML-based ones for sentiment analysis). Indeed, it is just confined to testing different prompting strategies, with two commercial models (i.e., GPT-4O and 4O-mini). Similarly, some experimental choices (e.g., the very low number of agents despite the sensitivity results) are not adequately motivated.
• The experimental results turn out to be particularly weak for 3 out of 7 tasks, with very low coding accuracies. Also, some additional quantitative measures (e.g., inter-agent agreement) would be beneficial for a better understanding of how SCALE handles the annotation processes.
• Despite aiming at fostering better human-AI interaction in content analysis, as well as strong capabilities, there is no human qualitative evaluation of the SCALE's capabilities. This would be needed to further validate the helpfulness of the proposed framework.
• The entire study relies solely on the GPT family of models. Experimenting with other (e.g., open) models would be beneficial for a broader applicability and adoption of the proposed framework.
• There are no technical details on the agents deployment and interaction. This is a key aspect for multi-agent systems, and should be stated in the manuscript to also foster reproducibility. Similar considerations hold for the human-in-the-loop setting.
• To properly validate how SCALE complements humans, there should be some more emphasis on the patterns occurring within it, and critical analysis on how the different phases differ or resemble humans. For instance, for RQ2, certain datasets see limited to no improvement after agents' discussion, why?

One big concern about the paper is that it does not provide prior benchmarks. The datasets used by the authors are not very commonly used in this literature. I recommend that the authors use at least one dataset with prior benchmarks on multi-label classification (e.g., COCO, Peskine et al. 2023) or apply prior methodologies of multi-label classification on your datasets. How does the plain-vanilla GPT-4o perform on your dataset without including multiple agents?

It is well-known that agentic approaches improve LLMs’ performance. However, the approaches typically require more computational resources and time. It would be helpful if the authors could include each ablation's cost and processing time. The authors acknowledge this issue in Section 6, but it will be helpful for the readers to see the informational gain of this framework along with computational requirements.

In Section 5.4.3, the authors might want to include some desired codebook structures in their prompt. They could add layer of agents that review the final product by including several instructions, e.g., examining whether there are overlapping categories by providing some theory backgrounds. They might even try fine-tuning the LLMs using some domain knowledge instead of using the plain-vanilla versions.

Missing citations: Several works have already explored how the discussion among LLM agents can improve overall performance. For example, see Chan et al. (2023) and Kim et al. (2024). I’m surprised that the authors do not acknowledge any of these studies. At a high level, what this paper shows is similar to the value of a multi-agentic approach in classification problems.

References Peskine, Youri, et al. "Definitions Matter: Guiding GPT for Multi-label Classification." Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. Chan, Chi-Min, et al. "Chateval: Towards better llm-based evaluators through multi-agent debate." arXiv preprint arXiv:2308.07201 (2023). Kim, Alex, Keonwoo Kim, and Sangwon Yoon. "DEBATE: Devil's Advocate-Based Assessment and Text Evaluation." arXiv preprint arXiv:2405.09935 (2024).

Minor Comments

1. You list five contributions but say the contributions are fourfold on page 2.
2. Why are some datasets benefiting heavily from discussions while others are not (Figure 4)? It would be helpful to include some insights on where the discussions will likely improve the model performance more and why.
3. In Table 3, it is concerning that you achieve the highest accuracy when human intervention is frequently made, and the LLM strictly follows it. Doesn’t this suggest that human interventions are required and LLMs alone cannot perform the task reliably?