SocraticLM: Exploring Socratic Personalized Teaching with Large Language Models

We sincerely appreciate your affirmation of our motivation, the contribution of our dataset, the novelty and generalization ability of our pipeline, and our sufficient experiments and comprehensive assessment.

$\bf{Q1}$:If GPT-4 itself cannot correctly solve a problem, can it still serve as a teacher in an educational role?

$\bf{A1}$:Thank you very much for your insightful question. We believe it still can serve as a teacher for two main reasons. Firstly, the Dean agent can judge and correct each round of the Teacher agent's instruction. Within a single round, the instruction needed to be judged usually involves just one reasoning step, making this process easier than having GPT-4 solve a problem directly. Therefore, even if GPT-4's problem-solving ability is limited, it still has the potential to judge and revise the instruction accurately. Secondly, our pipeline does not rely on GPT-4's inherent problem-solving ability because we input the correct solutions into the prompts for both the Dean and Teacher agents. In other words, our pipeline is indeed having SocraticLM teach the correct solution to students rather than solving the problem itself. Therefore, the problem-solving ability of GPT-4 does not limit the effectiveness of our pipeline. Your question is very thought-provoking, and we will supplement these discussions in the revised version of our paper.

$\bf{Q2}$:Why SocraticLM performs even better in problem-solving accuracy on MAWPS dataset?

$\bf{A2}$:Thanks for your valuable question. We think the reason is that from fine-tuning on our SocraTeach dataset, SocraticLM indeed learns to answer multiple questions from students about various aspects of a single problem (e.g., asking about each reasoning step and the involved knowledge). This process may allow SocraticLM to develop a deeper understanding of the problem-solving process, which in turn can improve its problem-solving accuracy.

We sincerely appreciate your affirmation of the significance of our motivation, the innovation and generalizability of our pipeline, the contributions of our evaluation system, and the effectiveness of our SocraticLM.

$\bf{Q1}$:How to assess the importance of Dean agent?

$\bf{A1}$:Many thanks for your careful comment. Yes, as you have noted, the effectiveness of Dean agent can be reflected by comparing the performance of SocraticLM and GPT-4. For a given preceding dialogue, the Dean agent judges and corrects the responses generated by the GPT-4 simulated Teacher agent. This is the fundamental root for the differences between GPT-4 and SocraticLM after fine-tuning. Therefore, the superior performance of SocraticLM compared to GPT-4 as shown in Table 1 can directly demonstrates that Dean agent is effective and necessary.

$\bf{Q2}$:Why the CARA metric of “w/o Correct” is lower than “w/o DiaS”

$\bf{A2}$:Thank you very much for the question. We think the reason is that the "w/o Correct" model was still fine-tuned on 10K single-round dialogues corresponding to students’ "Incorrect-reply" as explained in Section 3.4. This led the model to develop a stronger tendency to perceive a student's reply as incorrect, causing it to incorrectly classify some correct student replies as wrong. As a result, the model's performance on the CARA metric was worse compared to the model that did not include single-round data training (i.e., “w/o DiaS”). This phenomenon indicates that it is necessary to construct data for the "Correct-reply" category and balance it with the "Incorrect-reply" category.

$\bf{Q3}$:Exist some typos.

$\bf{A3}$:Thank you for pointing this out. We will carefully review and polish the writing of our paper.

Hello. This submission has been labeled by the reviewer for additional ethics review, I recommond the authors to priortize their response to the ethics concern so the ethics reviewers could get a clearer picture of the situation as early as possible. Please find above the ethics concerns raised by Reviewer sW5a.

The purpose of the additional ethics review is to help the authors mitigating potential ethical, legal or societal harm at an early stage (if there's any). Thank you for your understanding.

We sincerely appreciate your affirmation of the novelty of our SocraticLM, the value of our constructed teaching dialogue dataset, the innovation of our teaching ability evaluation system, and the clarity of our writing.

For your concerns regarding the ethics flag, we want to express our sincere gratitude for your attention and sorry for the misunderstanding. In this paper, we aim to construct a SocraticLM for personalized teaching. To achieve this, we first propose a “Dean-Teacher-Student” (DTS) pipeline to collect large-scale Socratic teaching dialogue data, where we implement all agents with GPT-4 and do not incorporate human subjects. As for our evaluation system, similar to other works, we invited 10 annotators to assess and compare the teaching instructions of different LLMs to test their performances. As shown in our annotation template in Appendix F, this process also does not involve research on humans or privacy/security risks. Following your suggestion, we will provide the necessary clarification and corrections to the conference and give a clearer introduction of our annotation process in the revised version. The following are the responses to your other questions:

$\bf{Q1}$:Standard errors or confidence intervals.

$\bf{A1}$:Thanks for your constructive comments. We add the following results:

$\bf{Q2}$:How this paper differs from and outperforms existing methods.

$\bf{A2}$:Thanks for your thought-provoking question. As summarized in our related work section, existing methods on Socratic teaching with LLMs can be divided into two categories. The first uses a general LLM (e.g., ChatGPT, GPT-4) to assist in conversation design in courses[1], content authoring[2], explaining learning-paths[3], and providing feedback[4]. Notably, the work you mentioned[5] uses GPT-4 to create an innovative Socratic Playground for Learning, constructing diverse learning scenarios where GPT-4 interacts with students in a Socratic manner. The second category collects data to train a specialized teaching LLM, with EduChat[6] being the most representative (also used as our baseline).

Compared to them, our SocraticLM improves from two main aspects. 1) When constructing the SocraTeach dataset, we build a student cognitive system to simulate six kinds of authentic students and enhance four key teaching abilities. This enables SocraticLM to handle more complex teaching scenarios and have more comprehensive and systematic teaching abilities. 2) Our DTS pipeline contains a novel Dean agent to judge and revise the GPT-4-based Teacher agent, addressing the limitations that existing general LLMs may still make for a bad teacher[7] and improving teaching quality. Experiments show that SocraticLM outperforms GPT-4 and EduChat across various dimensions, verifying the effectiveness of our pipeline and model.

Thank you for pointing out the excellent works and we will cite them in our revised paper.

[1] Chatgpt in the generalized intelligent framework for tutoring.

[2] Ruffle&riley: Towards the automated induction of conversational tutoring systems.

[3] Supporting student decisions on learning recommendations: An llmbased chatbot with knowledge graph contextualization for conversational explainability and mentoring.

[4] How can i get it right? using gpt to rephrase incorrect trainee responses.

[5] SPL: A Socratic Playground for Learning Powered by Large Language Model.

[6] Educhat: A large-scale language model-based chatbot system for intelligent education.

[7] The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues.

$\bf{Q3}$:Minor typos.

$\bf{A3}$:Thank you very much for thoroughly reading our paper and pointing out them. We will revise our paper carefully.

$\bf{Q4}$:Performance fall on some metrics at 125% data scale?

$\bf{A4}$:Many thanks for your insightful question. In Figure 4, at the 125% data scale, the metrics that decline are IARA and overall quality, indicating that the root cause is a decrease in the model's ability to identify incorrect answers (the decline in overall quality is a subsequent result). This may be because, with the increase in multi-round dialogue data, the proportion of single-round dialogue data for “Incorrect reply” decreases. When multi-round data scale exceeds 125%, the proportion of this single-round dialogue data may fall below a certain threshold, which results in diminishing effectiveness. Many thanks for pointing out this phenomenon. We will supplement these discussions and explanations in the revised paper.

$\bf{Q5}$:Will dataset be released?

$\bf{A5}$:Yes! We have already made the test set of our SocraTeach dataset and the model training code available through an anonymous repository (https://anonymous.4open.science/r/NeurIPS-4310). If our paper is accepted, we will release the full dataset as soon as possible.

$\bf{Q6}$:Current experiments are only in English and human evaluations come from a small sample.

$\bf{A6}$:Thanks for your constructive comments. Although our current dataset is in English, as mentioned in Section 3.2, our "Dean-Teacher-Student" pipeline is general and can easily be applied to other datasets, such as the Chinese Math23K dataset. For human evaluations, we sampled over 1,000 dialogues from SocraTeach dataset to calculate metrics and calculated the Kappa score within the annotators. The results demonstrate the consistency of the manual evaluations and the effectiveness of our SocraticLM. Inspired by your comments, we will consider constructing multilingual teaching datasets and training judging networks based on human-annotated results for larger-scale automatic testing in the future.

Many thanks for your response and increasing your score! We are very happy to address your concerns and will add all suggested clarifications and modifications into the revised version following your comments. Thank you again for the time you took to review our paper and your affirmation of our work. If there are any further questions, please feel free to raise them, and we can discuss any questions at any time. We will try our best and respond as soon as possible.

We sincerely appreciate your affirmation of the novelty of our pipeline, the contribution of our evaluation system, and the significance of our experiments.

$\bf{Q1}$:Compare with related works in evaluating LLMs for education?

$\bf{A1}$:Thanks for your valuable question. The related works can be divided into two categories. The first involves objective similarity assessments with human-annotated instruction, using metrics like BLEU and BERTScore[1]. The second involves subjective human evaluations, such as calculating the percentage where student reaches the correct answer given the instructions[1], analyzing the correlation between student grade changes and the use of LLMs[2,3], and issuing questionnaires[4].

Compared to them, our evaluation system offers several advantages:

1. More comprehensive and adequate assessment. It includes evaluations of overall teaching quality and four key teaching abilities, which current works lack a systematic organization.

2. Better comparability. Since a real student can only be taught by one LLM at a time, it is hard to compare the teaching effects of different LLMs by the aforementioned subjective human evaluations. In comparison, our evaluation is based on shared teaching dialogues, which can support the fair evaluation of multiple LLMs simultaneously.

3. More extensive. The latest dataset[1] contains only 600 testing dialogue data, which is far less than our data volume.

Frollowing your suggestions, we will supplement these discussions in our paper to highlight our contributions.

[1] A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems.

[2] Gptutor: a chatgpt-powered programming tool for code explanation.

[3] GPT-Empowered Personalized eLearning System for Programming Languages.

[4] Recipe: How to integrate chatgpt into efl writing education.

$\bf{Q2}$:Why allow the LLM to pick instead of specifying student cognitive states? The imbalanced distribution affects the performance?

$\bf{A2}$:Thanks for your insightful question. We allow the LLM to pick a student cognitive state for two reasons. First, real teaching processes are inherently complex, which may cover multiple cognitive dimensions (e.g., calculation, knowledge mastery) within a single teaching dialogue. Thus, manually specifying the state may limit the diversity and hinder the simulation of real teaching process. Second, it may disrupt the dialogue fluency. For example, if the Teacher agent asks "Do you know the concept of trigonometry?", it would be unnatural to ask the Student agent to response according to a weak "Problem Understanding" state, which may lead to an almost irrelevant reply, making it less like a real teacher-student conversation.

The imbalanced distribution will affect the model's performance. For instance, for a real student with weak "Instruction Understanding" state (i.e., type 2 in Figure 6), a model trained on an imbalanced dataset with much less data on type 2 may struggle to provide appropriate instructions. To address this, in Section 3.4, we employ data augmentation for four key teaching abilities, and the ablation study in Section 6.2 clearly verifies the necessity of this enhancement.

$\bf{Q3}$:Robustness when the data distribution changes (e.g., test on more MOOC question-answer pairs) and what about the dataset partition?

$\bf{A3}$:Thanks for your meticulous attention to our data collection. Firstly, sorry for our unclear statements. In Section 3.4, we collected 200 genuine student inquiries from MOOCs that are unrelated to teaching to conduct data augmentation for students’ “Irrelevant” responses, rather than collecting question-answer pairs. Secondly, we greatly appreciate your concern regarding the model robustness. As stated in lines 316-317 of Section 6, we remove the dialogues for questions in the evaluation set during training. Therefore, the issue of "the same questions appear simultaneously in both the evaluation set and the fine-tuning set" does not arise. This also addresses your point about the distribution shift or the need to test on new MOOC data, as our test set has not appeared in training, which validates the robustness of our model.

$\bf{Q4}$:Elaborate on metrics IARA, CARA, SER, and SRR.

$\bf{A4}$:Yes! These metrics are also evaluated manually. Since they are considered objective binary classification tasks as explained in Section 5, when annotating, we simply provide the preceding dialogue and ask annotators to evaluate whether the LLM's response "identifies the incorrect/correct student reply" (for IARA/CARA), "addresses the student's question" (for SER), or "refuses to answer the student's irrelevant question" (for SRR).

$\bf{Q5}$:Code for data synthesis.

$\bf{A5}$:Thanks for your comments. We supplement all the code of our agents needed for data synthesis in the folder “dataset_synthesis” in our repository. If the paper is accepted, we will also release all the data and code for public use as soon as possible.

$\bf{Q6}$:Evaluation that excludes the GSM8K questions that ChatGLM3 cannot be solved.

$\bf{A6}$:Thanks for your constructive suggestion. From the Table below, we observe that the performances of LLMs improve, but the increase is not particularly significant. We think this is because, in constructing our SocraTeach dataset, the prompts for Dean and Teacher agents include the correct solutions for the problems. In other words, we do not require ChatGLM3 to solve the problems directly but rather to learn how to teach the correct solution to a student. Thus, the problem-solving ability of ChatGLM3 does not set the upper limit for the teaching ability of our SocraticLM.