USDC: A Dataset of $\underline{U}$ser $\underline{S}$tance and $\underline{D}$ogmatism in Long $\underline{C}$onversations

We thank the reviewer for their strong positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1. I recommend that the authors revise these statements to reflect the results more precisely.

Thank you for pointing this out. We now revised the paragraph to better reflect the results as follows.

• Correction of Majority Voting Performance for Dogmatism Task under SLM fine-tuning:
  • Upon this clarification, for the Dogmatism Classification task, when using majority voting labels as ground truth, 2 out of 7 models in the fine-tuning setup achieve an F1-score above 50%. This distinction is now clearly reflected in the revised manuscript.
• Correction for line 407: “For both tasks when finetuning, the majority voting labels as ground truth has a relatively high performance, scoring above 50% weighted F1-score across several (7/7 for stance and 2/7 for dogmatism) models.”

Q2. why do SLMs fine-tuned with one-shot annotations outperform those fine-tuned with corresponding few-shot annotations? Could the authors provide a hypothesis that might explain this counterintuitive result?

Thank you for this question.

• When fine-tuning with few-shot annotations, the model might overfit to the small number of examples provided. This overfitting can lead to poorer generalization to new data, whereas one-shot annotations might strike a better balance between learning and generalization.
• For Mistral-large, we also observe this when computing inter-annotator agreement (IAA) with humans -- one-shot model has higher IAA compared to few-shot one.

We believe that the large size of GPT4 helps it to avoid this problem.

• Previous studies [Chen et al. 2023] have also shown that one-shot could be as good as or sometimes better compared to few-shot.

[Chen et al. 2023]. "How Many Demonstrations Do You Need for In-context Learning?." EMNLP Findings 2023.

We thank the reviewer for their strong positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1. What I am particularly concerned about is that you only used two LLMs for data annotation, which poses a risk of missing knowledge from other regions and fields, especially knowledge that other models might possess.

Thank you for this question.

• Before proceeding with LLM annotation using larger models, we first tested other versions of GPT, Mistral and LLaMA models, such as GPT-3.5 and Mistral-small and medium, LLaMA-2-70B. However, we found that these models failed to produce annotations in the desired format.
• GPT4 is known to perform better than most other models across various regions and fields. Hence, we believe that a combination of GPT4 and mistral-large was good enough given our limited budget.

Q2. I am also worried that in some cases, the system prompts may not be clear enough, leading to confusion in the large language models during annotation, such as inaccuracies in recognizing the author's stance.

Thank you for this question.

• The detailed system prompt for our LLM annotations is provided in Appendix C in the original paper.
• It clearly lists the objective of the task, definition of stance, definition of dogmatism, detailed task description. Next we also provide detailed definitions of each of the stance labels as well as each of the document dogmatism labels.
• Further we provide a detailed description of the different fields in the input data.
• Lastly we have also provided helpful instructions for effective annotation where rather than just requesting the model to assign the label, we also asked it to provide a concise justification.
• We therefore think that our system prompt is clear enough. Is there anything specific that you think we could have included?

Q3. Furthermore, the models face difficulties in identifying intermediate positions and ambiguous attitudes, which can easily lead to misunderstandings of user opinions. These issues will be even more pronounced in machine-annotated data, as you may not be able to provide sufficient information.

• For obtaining annotations, we provided the full conversation. Thus, we are able to provide sufficient information to the models (as much as we could provide to human annotators).
• In the recency bias experiments (lines 477 to 485), we observe that providing full context to models is better than providing just the recent context i.e. prior context. Hence, we provided full context to get our final annotations.
• Carefully designed system prompt, Few-shot prompting, majority voting are different ways in which we have attempted to obtain good quality annotations from LLMs.
• Lastly, GPT4 and mistral have significantly improved abilities to understand and maintain context over long conversations.

Q4. Please use more experiments to convince me that the weaknesses not exist.

• For weakness 1, we have already argued that GPT4 and mistral can handle most regions and fields. Also, we have limited budget.
• For weakness 2, we have already provided a clear system prompt.
• For weakness 3, we have already designed robust methods to ensure accurate labels.
  • We now include qualitative examples demonstrating cases with high, moderate, and low inter-annotator agreement (IAA) for the Stance and Dogmatism tasks, as shown in Appendix Figs. R.1, R.2, R.3, R.4, R.5 and R.6. In cases of high agreement, all LLMs consistently assign the same stance label to a user comment. For moderate agreement, some LLMs assign one stance class while others assign a neighboring stance class. For low agreement, GPT-4 assigns consistent stance labels across its three settings, but Mistral Large outputs differ for each setting. We have added these examples to Appendix R in the revised manuscript to illustrate where the labeling approach is most, moderate and least effective.

We thank the reviewer for their strong positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1. The majority voting conflict makes me wonder why Mistral is used at all if, in cases of conflict, the decision maker is GPT4 (which is quite a costly model)?

Thank you for this question. We would like to clarify the process of obtaining labels in our USDC dataset and the role of both GPT-4 and Mistral Large models in the majority voting setup.

• Label Distribution Across Models:

  • For the stance detection task, 85% of the labels (7969 out of 9324) were obtained through majority voting involving both GPT-4 and Mistral models. The remaining 15% (1355 labels) were derived from GPT-4 in cases where there was a conflict.
  • Similarly, for the dogmatism task, 88% of the labels (1349 out of 1528) were obtained via majority voting, while the remaining 12% (179 labels) were assigned based on GPT-4’s output in conflict scenarios.

• Why Both Models Are Necessary:

  • The majority of the labels (more than 85% across tasks) were determined through collaboration between GPT-4 and Mistral, making both models integral to the labeling process.
  • GPT-4 is only used as a tie-breaker in the minority of cases (15% for stance detection and 12% for dogmatism), ensuring consistency and reliability for ambiguous cases without over-reliance on a single model.

• Efficiency and Completeness:

  • This hybrid approach allows us to leverage the strengths of both models efficiently. Mistral provides cost-effective annotations for the majority of the data, while GPT-4 ensures high-quality resolution for edge cases, making the labeling process both scalable and accurate.

We hope this explanation clarifies the necessity of both models in our dataset annotation process.

Q2. Majority voting labels are used as ground-truth. It would be good to add experiments on what would happen if we train on unaggregated labels, as subjectivity is important in such a task.

Thank you for this question.

• We would like to clarify that our experiments are not limited to majority voting as ground truth. We have also performed fine-tuning and instruction tuning using zero-shot, one-shot, and few-shot labels from both GPT-4 and Mistral Large models as ground truth. This additional analysis provides a comprehensive evaluation of how different types of labels, including individual model labels, affect model performance.

• Analysis of Majority Voting vs. Individual Model Labels:

  • The results (Table 1 in main paper) demonstrate that majority voting labels consistently result in better model performance compared to labels obtained from individual models (e.g., GPT-4 or Mistral alone).
  • Majority voting helps mitigate noise and outlier annotations by capturing a more robust consensus, improving generalization and reducing variability during training.

• Unaggregated Labels Under Individual Model Performance:

  • The analysis of unaggregated labels is equivalent to conducting experiments with individual model labels. The performance using GPT-4 or Mistral labels alone reflects the impact of unaggregated, model-specific subjectivity.
  • These experiments highlight that while individual models capture useful task-specific features, aggregated majority voting labels lead to better alignment with the task’s ground truth.

In summary, our study already incorporates an analysis of unaggregated labels via experiments on individual model labels.

Q3. Why is it the case that instruction-tuning is better for stance and fine-tuning better for dogmatism?

Thank you for this thoughtful question. Below, we clarify why fine-tuning appears to outperform instruction-tuning for the dogmatism detection task and provide additional analysis to support our findings.

• Since dogmatism detection is inherently more complex and varied than stance detection, the model might struggle to generalize from the instructional data
• We report the confusion matrix for dogmatism detection task in Fig. 9 in the Appendix.
  • It shows significant misclassifications, especially for the ''Deeply Rooted'' and ''Flexible'' labels, with zero accuracy and F1-scores.
  • On the other hand, the model performs moderately better for ''Firm but Open'' and ''Open to Dialogue'' classes with accuracies of 48.7% and 64.4%, respectively.
  • The confusion matrix also indicates substantial confusion to distinguish between intermediate levels of dogmatism, such as ''Firm but Open'' and ''Open to Dialogue''.
  • This analysis demonstrates that dogmatism detection demands robust handling of long-term dependencies and task-specific adaptations, which fine-tuning achieves more effectively than instruction-tuning.

Q4. The observed inter-annotator agreement between LLM-human and human-human highlights the subjectivity of this task. If these labels are treated as ground truth in experiments, might this reduce the robustness of the results?

• Typical social media based datasets especially those involving long text are difficult to label objectively. That said, an inter-annotator agreement 0.49 for stance and 0.50 for dogmatism is quite reasonable. Previous studies have reported similar agreement values [Fast & Horvitz 2016] [Sakketou et al. 2022].

• Possible ways to improve IAA is to ensure that the annotation guidelines are as objective as possible. To ensure this, we already performed several iterations (both manually as well as using prompt optimization methods) and have carefully designed the annotation guidelines. We have used the same guidelines as part of the prompt. The prompts are already included in the Appendix.

• Using this dataset, we train llama-based models. These models are known to be robust to training label noise.

• The IAA is around 0.5 but it only captures mismatches and not the degree of mismatch. In our case, the classes have a ranking order. The predictions from human vs LLM are close to each other in this order even when they mismatch.

• Appendix Q Fig. 21 presents a heatmap comparing human-annotated labels and majority voting labels from LLMs, illustrating the class-specific agreement for Stance and Dogmatism tasks. From Fig. 21 (left), we make the following observations:

  • (i) The ''Stance Not Inferrable'' (SNI) and ''Strongly Against'' (SGA) classes exhibit high agreement between human annotations and LLM predictions, as indicated by the strong diagonal values for these categories. These classes achieve recall values of 0.97 and 0.65, respectively, when compared to human labels.
  • (ii) ''Somewhat in Favor'' (SIF) and ''Somewhat Against'' (SOA) show substantial mismatches with human labels, leading to higher rates of false positives in LLM predictions.
  • (iii) Notably, ''Somewhat Against'' (SOA) demonstrates the greatest level of disagreement (recall of 0.37), with frequent misclassification into neighboring categories such as ''Strongly Against'' (SGA) or ''Somewhat in Favor'' (SIF).

• For Dogmatism task, we make following observations from Fig. 21 (right):

  • (i) The ''Firm but Open'' (FBO) and ''Open to Dialogue'' (OTD) classes exhibit relatively high agreement, with strong diagonal values in the confusion matrix. These classes show better alignment between human labels and LLM predictions compared to other dogmatism categories.
  • (ii) The ''Deeply Rooted'' (DR) and ''Flexible'' (FX) classes have significantly fewer samples and exhibit frequent misclassifications. For instance, ''Deeply Rooted’’ (DR) is often misclassified as ''Firm but Open’’ (FBO), indicating challenges in detecting extreme levels of dogmatism.

• Overall, the significant mismatch for intermediate stance classes, particularly ''Somewhat Against’’ in the stance detection task and ''Open to Dialogue’’ in the dogmatism task, likely explains the moderate inter-annotator agreement (IAA) observed between human and LLM-generated labels.

We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1. Moderate Inter-Annotator Agreement: The inter-annotator agreement between human and LLM annotations could be improved.

Thank you for this question.

• Typical social media based datasets especially those involving long text are difficult to label objectively. That said, an inter-annotator agreement 0.49 for stance and 0.50 for dogmatism is quite reasonable. Previous studies have reported similar agreement values [Fast & Horvitz 2016] [Sakketou et al. 2022].
• Possible ways to improve IAA is to ensure that the annotation guidelines are as objective as possible. To ensure this, we already performed several iterations (both manually as well as using prompt optimization methods) and have carefully designed the annotation guidelines. We have used the same guidelines as part of the prompt. The prompts are already included in the Appendix.

[Fast and Horvitz 2016], "Identifying dogmatism in social media: Signals and models." EMNLP (2016)

[Sakketou et al. 2022], "Investigating user radicalization: A novel dataset for identifying fine-grained temporal shifts in opinion." arXiv preprint arXiv:2204.10190 (2022). https://arxiv.org/pdf/2204.10190

Q2. Fragmented Conversations: By selecting only the top two authors’ comments, the dataset lacks conversational continuity. The two selected authors’ comments are scattered across the thread, rather than forming a cohesive conversation.

Thank you for raising this important question.

• We would like to clarify that although we annotated stance and dogmatism labels for the top two authors, we included the full conversation thread with all user comments in our dataset. This ensures that there are no fragmented conversations, as the full context of every user comment is provided.

• The dataset captures the entire thread, allowing the LLM to analyze how a user changes their opinion based on other users' comments within the context of the target conversation. This setup enables us to study how a user’s opinion evolves in response to other users’ comments throughout the entire conversation.

• Our decision to focus on the two most active users stems from the observation that users with fewer comments often do not provide sufficient data to accurately assess their stance or dogmatism. Many users contribute only one or two comments, which is typically insufficient for determining their overall opinion or dogmatic nature.

• By prioritizing the two most active users—who contribute approximately 50% of the comments in each conversation—we ensure that our analysis captures meaningful opinion fluctuations and provides a robust evaluation of stance and dogmatism. Opinion fluctuations, as well as stance and dogmatism detection can be useful to build moderation tools, which will be most applicable for active users.

• While we acknowledge that including additional users could enhance the dataset’s completeness, we believe our current approach strikes a balance between computational feasibility, analytical depth and practical usability. We appreciate the reviewer’s insight and will clarify this aspect further in the revised manuscript.

Q3. Are there any qualitative examples demonstrating cases with high, moderate, and low inter-annotator agreement? These could provide insight into where the labeling approach is most and least effective.

Thank you for this question.

• Based on the reviewer’s suggestion, we now include qualitative examples demonstrating cases with high, moderate, and low inter-annotator agreement (IAA) for the Stance and Dogmatism tasks, as shown in Appendix Figs. R.1, R.2, R.3, R.4, R.5 and R.6.
  • In cases of high agreement, all LLMs consistently assign the same stance label to a user comment.
  • For moderate agreement, some LLMs assign one stance class while others assign a neighboring stance class.
  • For low agreement, GPT-4 assigns consistent stance labels across its three settings, but Mistral Large outputs differ for each setting.
• We have added these examples to Appendix R in the revised manuscript to illustrate where the labeling approach is most, moderate and least effective.