Does Collaborative Human–LM Dialogue Generation Help Information Extraction from Human–Human Dialogues?

We thank the reviewer for finding our human-LM collaborative framework interesting.

We proposed a new IE metric because the DST metrics, joint goal accuracy (JGA) and slot accuracy, can not well reflect the model performance on our task. Here are more details.

First, JGA is chosen because DST tasks require database queries; hence accurate prediction is needed. Queries are issued after most turns, so evaluating average performance at all turns makes sense. In contrast, in our IE task, where information is accumulated (and corrected) for a final summary. Our goal in evaluating car accident claims is to assess the overall extraction quality. Minor errors (e.g., additional word in non-categorical slots) should not significantly impact the readability of the extracted information. Therefore, we chose more flexible F1 metrics.

Second, both slot accuracy and our IE metric exhibit the same trend where later turns receive lower scores. It is because the later turns are harder to predict because more information is presented in a dialogue. Slot accuracy is computed at each turn and defined as:

$\text{Slot accuracy} = \frac{T - M - W}{T}$,

T: # all predefined slots. M: # missed slots that the model does not predict among the gold slots. W: # wrongly predicted slots that do not exist in the gold slots.

Early turns usually contain a few gold slots, resulting in high scores for every model and making the performance difference between models become trivial. In contrast, our IE metric evaluates at specific dialogue points (quarter, half, three-quarters, end), mitigating the issue of misevaluation of models at earlier turns.

Regarding the use of powerful models, we chose finetuned T5 models for two reasons: (1) previous work [1] shows that it outperforms large decoder-only models in finetuned domains, and (2) our resources for finetuning on private data are limited, preventing the use of larger models.

Moreover, in this work, we explored the experiments with the in-context learning (IC-DST w/ ChatGPT) on the synthesized data due to the privacy concern. We find that IC-DST performs worse than a finetuned T5 up to 12% as shown in Table 2. However, we acknowledge that it could be interesting to explore recent open models (e.g. LLaMA3) on the private data in the future if resources permit.

[1] E. Lehman, E. Hernandez, D. Mahajan, J. Wulff, M. J. Smith, Z. Ziegler, D. Nadler, P. Szolovits, A. Johnson, and E. Alsentzer. Do we still need clinical language models?

We thank the reviewer for finding our work impactful in contributing to the research field.

Regarding reproducing results on a public dataset, we acknowledge that the additional results on a public dataset would increase the reproducibility of this work. However, assessing impact on real, complex, goal-oriented conversations is a key objective of our work, and it is challenging to find a public dataset that contains realistic dialogues due to the privacy issue.

We follow previous work (IC-DST) to represent the ontology via a SQL table format. In Appendix D3, we show an example of the prompt. For example, the domain Adjuster (as shown in the following text block) contains the domain slots Explain Coverages, Permission to Record, Set up Inspection, Set up Rental. The domain slots with CHECK values indicate they are categorical slots. Otherwise, the slots are non-categorical. We will update our description to clarify the ontology representation format.

CREATE TABLE Adjuster(
'Explain Coverages' TEXT,
'Permission to Record' TEXT CHECK ('Permission to Record' IN 'Yes', 'No'),
'Set up Inspection' TEXT CHECK ('Set up Inspection' IN 'Quick Photo Claim', 'Field
Assignment'),
'Set up Rental' TEXT CHECK ('Set up Rental' IN 'Yes', 'No'),
)

Thank you for finding that our methodology is impactful in contributing to the discussion around language model (LM) performance and showing that the current LM is still limited without human correction in data synthesis and annotation. We acknowledge that using open-source LMs would increase the reproducibility and enhance our findings. We used these private models as of the paper submission because they are the current state-of-the-art models in this field. Recent open LMs, such as LLaMA3 (released after this work), could provide an alternative LM for our human-in-the-loop framework. We leave it as a future work to explore.

We acknowledge the raised issue of reproducibility of using LMs. Based on recommendations from OpenAI, the temperature value ranges from 0 to 2, with lower values indicating greater determinism and higher values indicating more randomness. In our quantitative experiments, only the IC-DST experiment involves a private LM. To enhance the reproducibility of the IC-DST experiment on the publicly available DialGen-AIC, we set the temperature to 0 and report the API version (e.g., gpt-3.5-turbo-0301).

Thank you for your suggestion about the footnote. We will edit the paper and make the footnote appear at the first-mentioned position in the main content. Regarding the readability suggestion about the version of API calls of language models, we added a table to indicate the version of different LMs in Appendix C.2 of page 16 due to the page limit. We will add the SBERT and T5 into the table and add a reminder to inform readers the table is in the appendix.

We thank the reviewer for finding our human-corrected synthetic data approach interesting.

Regarding more thorough methods to reduce discrepancy between synthetic and real data: We showed that DialGen is capable of synthesizing dialogue data that is more similar to real data (AIC) than fully automatic data. However, our analysis (section 6.1) also shows there are still gaps, consistent with the reviewer’s comment. We acknowledge that future work can explore additional methods to further minimize the difference between the synthetic and natural data.

Regarding the effect of using real data alongside the generated data with the same amount, the result can be inferred from Figure 4. In the figure, we use 7 real dialogues as a training set (1.3K turns) and obtain a 34.8 TLB score. Using the same amount of generated data should be in the middle of 10% (0.9K) and 20% (1.8K) in the x-axis, suggesting the TLB score of 1.3K-turn generated data will be between 27 - 31. We can add an experiment to confirm the inferred number.

Regarding the missing statistical analysis of the results, we assume the reviewer is suggesting an additional statistical difference test between the experiments. We conducted the statistical difference test on our best model (T5-SC), as shown in Table 3. We used the bootstrap test, recommended in prior research [1][2], to evaluate the difference between the two results using the proposed $CB_4$ F1 metrics. We compared our best T5-SC result (AIC + DIALGEN-AIC) with the T5-SC result using only AIC. The resulting p-value < 0.01 indicates that incorporating both DIALGEN-AIC and AIC as training data is significantly better than using AIC alone.

[1] Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An Empirical Investigation of Statistical Significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995–1005, Jeju Island, Korea. Association for Computational Linguistics.

[2] Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, Australia. Association for Computational Linguistics.