RedHat: Towards Reducing Hallucination in Essay Critiques with Large Language Models

Q1: hallucination definition. Whether an LLM fails to focus on the desired argument or is too focused on the conclusion, is it fit to be called a hallucination?

As we have discussed in Section 3. The core of essay critiques can be divided into hallucinating non-existing writing problems and overlooking the essay's logic and structure with overemphasizing details, or vague comments with no in-depth advice.

We discuss the above mentioned question in two cases:

• Not hallucination: if the desired argument is important in the logic frame of the essay, such focus is needed and the supporting evidence and logical structure should be inspected before LLMs come to a final critique.
• Misunderstand the author’s perspective (Type I hallucination): if one perspective is first introduced in the introduction (opening) of the essay, the author is not expected to expand it with arguments - in other words, LLM fails to understand the author's intention or perspective if it is focusing on the arguments.

We have also made a comparison with our definition of hallucination with previous works in summarization, knowledge QA and reasoning.

Q2: Stability of the questions with each run of model

Thank you for your reminder. We repeated using GPT-4o to generate a question list from the evaluation criteria. We conducted 5 trials, asking GPT-4o to generate 20 question-answers each time. We compared the latter 4 trials's recall rates to the first trial as consistency listed below. Most of the questions are the same and it validates stability.

We greatly appreciate your reviews and valuable suggestions. We have carefully addressed each of your concerns in detail.

W1-1: reliability of GPT and human annotations

Thanks for your comments. We have added Qwen-2-7b-Instruct on EssayC along with ChatGPT-4o on a subset of artificial intelligence conference papers. Please refer to General Response 1 for details. In the Qwen experiment, humans are more inclined to believe that RedHat has improved the critiques' quality (around 7-10% more preferred on RedHat).

Sorry for not being clear enough with the previous performance presentation. We sorted out the experiment data, including Redhat is good, baseline-LLM is good, both are good and both are bad. We found that the performance of RedHat is more preferred by humans compared to baseline-LLM in both Qwen and GLM. Therefore, we believe that Redhat made improvements when judged by both humans and GPT-4. We have redrawn Figure 4, split the contents of the tie, and added Qwen’s experiment results.

W2-1: why word-level overlap metrics & consider BLEURT and BERTScore

Thank you for your valuable insight. We have chosen ROUGE and BLEU to verify that the embedded questions do not show leakage of desirable contents in explicit n-gram forms. Our results also verify that the questions are not similar to the final RedHat-generated critiques at the word level.

We have added BLEURT and BERTScore as similarity scores and updated them in the revised version. We quote the results in the tables below.

• The similarity between the critique with the questions (column 1) and the answers (column 2) are presented. Comparing the baseline (Row 1) with RedHat (Rows 2-3), there is no significant change in the similarity of the question, but the critique generated by RedHat is clearly more similar to the answer. This confirms that the gain in evaluation is benefited from the answer inferred by the LLMs themselves, rather than by the criteria-derived question. We have corrected this confusion in our revised version.

W2-2: paper writing that Chapters 2, 3, and 4 each contain a literature review with the work of this paper.

We sincerely appreciate your suggestions. We have moved the Related Work section to the appendix. Specifically, the second section formally defines the essay critiquing task, the third section illustrates hallucinations in our task, and the fourth section introduces our method to mitigate hallucinations. At the beginning of each paragraph, we have briefly described its content, and we have also added a description of the paper's structure in the introduction section to improve readability.

W3: annotation process and quality assessment of the data resource

Thanks for your suggestion. Please refer to General Response 2 for the detailed description of EssayC. We have introduced the annotation process and the quality assessment in the Filter and Annotation and quality assessment part.

Dear Reviewer rgUd,

The authors are grateful for your insightful comments. Since there are three days left until the discussion period ends, we are unsure whether the response above has fully addressed your concerns. If there are any remaining issues, please feel free to contact us. We sincerely look forward to your feedback.

Best regards,

The authors

We deeply appreciate your recommendation of our work and your advise to compute the Cohen's Kappa as indicator of inter-annotator agreement (IAA for short).

We have calculated the IAA for each number from the main experiment annotation results and have listed in the following table. We calculate the initial annotation agreements between two annotators before the introduction of the third annotator to recheck the disagreed results. The results are listed in the following table. Most of the agreements fall in intervals [0.4, 0.6) and [0.6, 0.8), i.e. between moderate/substantial agreements. The overall score's IAA is generally lower than dimensional ones, since it has up to 6 categories. The dimensional scores are mainly substantially significant. Results indicate that most annotators agree with each other in the first round of double check and there are certain levels of subjectivity during the process.

Thank you for author's response! I think what the authors are missing is not how humans assess the performance of these models, but rather whether the work of the humans is reliable. You should calculate metrics like Cohen's kappa to evaluate the consistency of the human annotators' work.

That said, I believe the authors have addressed most of my concerns, and I hope they can consider incorporating this metric in the future. I have decided to raising my score.

Q2: baseline methods information on training data for SFT and PT

We apologize for the confusion and inconvenience. We have made revisions to the manuscripts and briefly introduce the training data of supervised finetuning (SFT) and post-pretraining (PT) as follows.

For SFT, we adopt 31,694 polished human paragraph-level critiques as training data, excluded from the EssayC testset split (please refer to General Response 2 for more details). The format of the data is arranged into (eval prompt, essay, and target paragraph) as input, and polished paragraph as output. The train and valid set are split based on essays to avoid potential leakage.

As for post-pretraining, we follow two steps: (1) pre-training on Chinese academic papers in the field of literature, social science and humanities and (2) followed by SFT on the previous data to ensure the alignment of the critiquing task. We crawled 128,321 academic papers from the Chinese National Social Science Base. The papers mainly come from journals, such as Exploration and Free Views, Fiction Monthly Shanghai Literature, Beijing Literature Novella Month, Science Technology Critiques, Tanzhen Technology Review and so on. We use OCR with doc2x API (https://v2.doc2x.noedgeai.com) and applied the follow-up data filter and typo fixing with GPT and GLMs. The above process produces 27,430 pure text papers of an average of 30,000 Chinese characters. The whole tokens surpassed 1.5 billion.

Q3: validate the few-shot approach on this task

We sincerely appreciate your reminder of the few-shot methods. We conducted a few-shot (5-shots) experiment in the Qwen-2 experiments and the results can be found in General Response 1. We highlight the comparison with RedHat below. The results showed that 5-shots exhibited positive improvements in the 4 metrics, and the improvements are more evident. Yet, there is a gap between both of the two methods and human critiques.

Q4: Renaming "Detailedness" to "Ambiguity"

We sincerely appreciate your suggestion. In Section 2 we change the objective function to minimize ambiguity, as well as the changed the notation of D(c) to A(c). Also, we rephrase the "Detailedness" into "Ambiguity" in all experiment result tables.

Thank you for your thorough review and valuable feedback! We have worked to address your concerns one by one.

W2: unclear whether the issue the authors address is truly "hallucination" & types of errors not regarded as hallucination

In our work, LLM-generated essay critiques mainly face two types of problems. Type I: Hallucinate writing problems that do not exist in the essay. Type II: Overlook the essay's logic and structure by overemphasizing on details, or just vague comments with no in-depth advice. We find the two types are consistent with hallucinations discussed in previous literatures. Specifically speaking,

• For type I, [1] mentioned "ignore the source material altogether" as Extrinsic Hallucination in summarization task. [2] defined "generation of nonfactual, untruthful information, especially ignoring the context" in ChatGPT's reasoning and multimodal tasks.
• For type II, [1] mentioned "misrepresent information from the doc" as Instrinsic Hallucination in summarization task, [3] concluded "Generation of plausible looking statements that are factually incorrect" in conversation task.

Considering the features of our type I and II hallucinations, our definition agrees with hallucination definition in summarization, conversation, QA, reasoning tasks. Based on the discussion above, we lend the wording hallucination to describe our phenomenon. We present the discussed literature in the following Table 5 for further interest.

In our work, errors that do not relate to the factuality of the original essays are not regarded as hallucinations. Here are two examples: (1) segmentation, that the critique is not presented in complete sentences; (2) tone or roleplay, whether the critique is expressed in the tone of an instructor or an assistant. They are beyond the scope of our work and also do not dominate in practice, and we did not include them in our category.

[1] Joshua Maynez et al. On faithfulness and factuality in abstractive summarization. ACL. 2020

[2] Kurt Shuster et al. Retrieval augmentation reduces hallucination in conversation. EMNLP. 2021

[3] Yejin Bang et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. IJCNLP. 2023

W1: The proposed method, RedHat, simply adds document-level questions and answers, which may come across as an incremental, somewhat trivial improvement.

We appreciate the reviewer's comments and valuable suggestions. As we discussed in Section 3, LLM-generated essay critiques mainly face two types of problems.

• Type I: Hallucinating writing problems that do not exist in the essay.
• Type II: Overlook the essay's logic and structure by overemphasizing on details, or just vague comments with no in-depth advice.

We identify them as hallucinations as [1], [2], [3] did in summarization, knowledge conversation, QA and reasoning tasks. Yet there is a gap in research addressing essay critique generation, especially prompt engineering, like adjusting the order of presenting inputs, few-shot or in-context learning, SFT method cannot reduce hallucination, indicating critique hallucination is not caused by task mis-alignment. We discover that it is closely related to the un-groundness of LLM to the target essay.

We have curated RedHat, aiming at improving LLM's awareness of the target essay's perspectives and logic structure. On the one hand, RedHat facilitates the assessment process into (1) proposing questions crucial for evaluating the essay; (2) answering questions based on the evidence from the essay, and (3) feeding the above digest to the evaluation instruction. RedHat harnessed LLM's ability to improve its awareness of the details and logical structure of the essay. On the contrary, SFT methods on the polished human critiques result in heavy hallucination (10% drop in hallucination in the main experiment), and few-shot methods have less effectiveness in results compared to RedHat.

To further prove our solution's effectiveness, we added experiments with Qwen-2-7b-Instruct with EssayC, and ChatGPT-4o with Computer Science Conference papers (mainly ICLR, Neurips, ACL). Results are shown in General Response 1. RedHat shows its effectiveness in both experiments, both in point-wise scoring and preference comparison. This also indicates the generalizability of our methods to more models and writing genres.

[1] Joshua Maynez et al. On faithfulness and factuality in abstractive summarization. ACL. 2020

[2] Kurt Shuster et al. Retrieval augmentation reduces hallucination in conversation. EMNLP. 2021

[3] Yejin Bang et al. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. IJCNLP. 2023

W3 & Q1: Vague on dataset & whether the findings apply to other datasets, languages, or base LLMs

We appreciate your suggestion. We have added the description of the EssayC dataset in Section 2 to echo the details in the revised paper. Please refer to General Response 2 for more information about EssayC.

We have also added experiments with Qwen-2-7b-Instruct on EssayC as well as with ChatGPT-4o on artificial intelligence conference papers. The details can be found in General Response 1. Briefly speaking, results from the Qwen-2 model on EssayC and from ChatGPT-4o on the English artificial intelligence conference paper are consistent with the previous experiment. (GLM-4 on EssayC), which demonstrates that our findings can be applied to further datasets, languages and base LLMs.

Q1: Will you open-source the dataset?

If accepted, we will make the dataset and manual annotation results public based on the wishes of the undergraduate essay authors and graduate teaching assistant annotators. We regard the resource having the potential to develop automatic evaluation and alignment methods for automatic essay critique generation.

Dear Reviewer ssM2,

The authors appreciate your efforts in improving the manuscript. Since there are three days left until the discussion period ends, we are unsure whether the responses above has fully addressed your concerns. If there are any remaining issues, please feel free to contact us. We are looking forward to your reply.

Best regards,

The authors

Dear Reviewer,

We are deeply grateful for your devotion to reviewing our manuscript. We have added two further experiments on different base-LLMs (Qwen2 and GPT) and languages (English). The extra experimental statistical results have also indicated the effectiveness of RedHat. We also have made clarification on concept definition, training data construction, the mechanism of our methodology in reducing the hallucination in critiques and other details. Hope our updated results resolve your concerns.

As this may be our final opportunity to address any remaining concerns you may have, if there are any aspects of our paper that require further clarification, please don't hesitate to inform us. We eagerly await your insights and the opportunity to engage in a constructive discussion to strengthen our submission. Thank you!

Best regards,

The authors

Minor concerns 2: strong vocabulary for contributions

We apologize for the inaccurate statement. In our revised version, we have corrected it to "suggest" and added the Qwen-2-7b experiment on EssayC as well as English experiments on ChatGPT-4o. Our results demonstrate that our approach is effective in: (a) different base models (GLM4, Qwen2, ChatGPT-4o), (b) the same base model with and without SFT, and (c) different types of long writings and languages such as undergraduate essays and conference papers.

Also, we kindly remind the reviewer that in Table 4, the horizontal axis represents the evaluators (GPT4 for auto method, human for manual method) of the quality of the polished paragraphs. The number ~60% reveals that GPT evaluates them as either equally good or equally bad. We conclude that both humans and GPT prefer the polished paragraphs based on RedHat critique.

Recommendations (a): root more firmly in existing literatures

We are very grateful to follow your kind and detailed suggestions, which essentially improved the quality of our work. We concluded the corresponding improvements to them.

We summarize the aforementioned literature discussion and included it in the revised version. [1] directly provides a score without critiques, hard to improve a 5000 characters essay; [2] offers critiques that do not address the improvement details, lacking paragraph-level feedback and compromising evaluation; [3] proposes topics, argumentations and responses in criteria for argumentation writing in education, and our evaluation criteria and derived questions are not conflicting with it. The above work's list of evaluation criteria differs from our work for the differences between essays and conference paper purposes. We have also added a few-shot method as a practicable baseline to our experiments. Thank you for your suggestion; we have added the few-shot method to the main experiment table.

[1] Xiaoyi Tang et al. Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments. Heliyon. 2024.

[2] Ryan Liu, Nihar B. Shah. ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing. Arxiv. 2023.

[3] Omid Noroozi et al. Design, implementation, and evaluation of an online supported peer feedback module to enhance students’ argumentative essay quality. Education and Information Technologies. 2023.

Recommendations part (b): increase the scope

We have added the results of experiments of Qwen2-7b on EssayC and ChatGPT-4o on English Conference papers. RedHat outperforms the baseline, SFT, few-shot, etc., in overall performance, hallucination, ambiguity, informativeness, and pairwise comparison. This demonstrates that RedHat reduces hallucinations and improves the credibility of the critiques. We have also updated the experiment section to include these results in our revised manuscript.

To automatically score overall/hallucination/ambiguity/informativeness, we have trained 4 Linear Regression Models by using the GLM4 and Qwen2 experimental data as a score predictor for response quality. We made a random split between the training set and the validation set with a ratio of 9:1. The regression fitting accuracies are shown in the table below. Results showed that the fitting in the hallucination dimension is not good, indicating our view that SFT alone does not contribute to hallucination relief, and echoing the issues in our paper. Thanks for your suggestion and we will release this score model and consider using RLHF in future work to further improve scoring quality.

W3: regarding the impact: task of essay feedback generation, studied language and generalising the findings

Thanks for your suggestion.

• On the one hand, we additionally applied our method to Qwen-2-7b-Instruct on EssayC and ChatGPT-4o and the task of English Artificial Intelligence Top Conference Paper review. Please refer to General Response 1 for result details and General Response 2 for data collection information. Results showed an apparent increase in credibility in RedHat-assisted critiques compared to the Qwen-2 and ChatGPT-4o baselines. This verified the generalizability of RedHat on English domain writing genres and different scaled LLMs.
• On the other hand, in words of the contribution of our work:
  • Firstly, our work has clearly defined the task of fine-grained essay critique generation, and also curated a testset with manually annotated quality results. This can serve as a benchmark for this field, as well as resources for automated evaluation development.
  • Secondly, in terms of methodology, by leveraging the LLMs' strong QA capabilities, our approach avoids reliance on complex reasoning or simple few-shot prompt engineering. Our method outperforms SFT and post-pretraining approaches, and reveals that task-specific SFT actually introduces more hallucinations. These also validate the hallucination problem is distinct from alignment or under-fitting ones.
  • Lastly, in terms of effects compared to previous works, we have pioneered fine-grained critiques for essays. Such critique generation methods for long essays or papers and critique quality evaluation are not discussed in previous work. Our approach is also effective across open-source 7B and 9B models as well as larger API models. In evaluating generation quality, we provide both pointwise scoring and pairwise comparison, which previous works have not yet focused on.

Minor concerns 1: the wording of hallucination

In our work, LLM-generated essay critiques mainly face two types of problems. Type I: Hallucinate Fantasizing writing problems that do not exist in the essay. Type II: Overlook the essay's logic and structure by overemphasizing on details, or just vague comments with no in-depth advice. We find the two types are consistent with hallucinations discussed in previous literatures. Specifically speaking,

• For type I, [1] mentioned "ignore the source material altogether" as Extrinsic Hallucination in summarization task. [2] defined "generation of nonfactual, untruthful information, especially ignoring the context" in ChatGPT's reasoning and multimodal tasks.
• For type II, [1] mentioned "misrepresent information from the doc" as Instrinsic Hallucination in summarization task, [3] concluded "Generation of plausible looking statements that are factually incorrect" in conversation task.

Considering the features of our type I and II hallucinations, our definition agrees with hallucination definition in summarization, conversation, QA, reasoning tasks. Based on the discussion above, we lend the wording hallucination to describe our phenomenon. We present the discussed literature in the following Table 5 for further interest.

[1] Joshua Maynez et al. On faithfulness and factuality in abstractive summarization. ACL. 2020

[2] Kurt Shuster et al. Retrieval augmentation reduces hallucination in conversation. EMNLP. 2021

[3] Yejin Bang et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. IJCNLP. 2023

• We would kindly remind the reviewer that our criteria posted in Appendix Section B matches the widely acknowledged standards for argumentative writing, which mainly focuses on topics, argumentatives, structures, literature and expressions. For example, [3] also proposed topics, argumentations and responses in criteria for argumentation writing in education for 800 words English essay. Moreover, we manually classify the questions into the five categories and the results are shown in the following table, which are not basically conflicting with that one acknowledged in education. The authors hope to address your concerns.

[1] Xiaoyi Tang et al. Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments. Heliyon. 2024.

[2] Ryan Liu, Nihar B. Shah. ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing. Arxiv. 2023.

[3] Omid Noroozi et al. Design, implementation, and evaluation of an online supported peer feedback module to enhance students’ argumentative essay quality. Education and Information Technologies. 2023.

W2: regarding the results are not very impressive and analysis of experiments

Thank you for your suggestions regarding the methodology. We have added experiments based on Qwen2-7b-instruct on EssayC. As described in the General Response 1, RedHat outperforms the Baseline by 3.8%, 6.585%, 6.83%, and 0.136 (Hallu, Ambiguity, Info, Overall), and surpasses SFT by 12.4%, 20.00%, 34.94%, and 0.652. Additionally, the QA accuracy of Qwen reaches 92.3%. We will continue to compare with LLaMA-3.1 in English experiments, which will be included in the final version of our paper.

W2-1: small improvement

The "Overall scores" represent the critique credibility, specifically their hallucinations. To clarify the effectiveness of our work, we have updated the polished human critique scores with the same annotation standard. The human overall score outperforms the best result (GLM+RedHat) by 0.06, it outperforms from the baseline(GLM) by 0.137, while directly SFT with revised instructor critiques undermines the overall score(-0.57). Thus, without introducing additional hallucinations, RedHat is closer to the level of human comments.

W2-2: not convinced by section 5.3

Sorry for the confusion. We would like to clarify that there is no inherent connection between the generated critiques and the criteria-derived questions. We have adopted Reviewer rgUd's suggestion and updated the BLEURT metric as the similarity algorithm. The results are shown in the Table below. The similarity between the critique with the questions (column 1) and the answers (column 2) are presented. Comparing the baseline (Row 1) with RedHat (Rows 2-3), there is no significant change in the similarity of the question, but the critique generated by RedHat is clearly closer to the answer. This confirms that the gain in evaluation is driven by the answer inferred by the LLMs themselves, rather than by the criteria-derived question. We have corrected this confusion in our revised version.

W2-3: the scale of the dataset is small and findings are not statistically significant.

We would like to remind the reviewers that our task is to generate paragraph-level critiques for essays, and we collected 395 paragraphs from 36 essays in EssayC to be inferred by LLMs. Actually, among works related to automatic essay reviewing, works from [2] have not surpassed the number of 200 samples. The comparison with previous work can be seen in the following table. When turning to the LLM alignment dataset, datasets with fewer samples have also received wide concerns like MT-Bench(80 samples, with 2 turns, 160 in total).

As for statistical significance, we did the T-test on GLM-4 and Qwen-2 experiment overall scoring on baseline-LLM and RedHat, and got p equals to 0.048 and 0.043, indicating statistical significance to the changes.

[2] Ryan Liu, Nihar B. Shah. ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing. Arxiv. 2023.

We sincerely appreciate the reviewer's devotion to reviewing our manuscript and the detailed recommendations for improvement. We address the reviewer's concerns as follows and hope the response can address your concerns.

W1: regarding existing literature on automatic essay evaluation, and human written critiques.

Thanks for your comments. The core contribution of our work focuses on fine-grained essay critique generation, revealing LLM-generated critique credibility problems such as un-groundness on target essay, fantasization of non-existing problems, especially suggesting adding excessive local details, showing that LLM overlooked the main arguments and logical flow. We summarized them as hallucinations. On the basis of un-groundness and overlooking essay contents in LLM's performance, we propose RedHat to reduce the hallucination of LLM-generated critiques. Specifically:

• Although LLMs are widely used for evaluation, they still face challenges in the credibility of critiques for essay reviews. We have surveyed existing work in automated essay evaluation, summarized in Table 1. Current studies either focus on scoring essays (a regression task, Row 3) or lack detailed critiques (mainly summarization of the writing, Row 4-5). Moreover, the inaccessibility of fine-grained essay critique benchmarks makes it difficult to address the reliability issue.

• RedHat aims to guide self-QA based on evaluation criteria for essay writing, encouraging LLM self-reflection in the generation process to address the neglect of target essay contexts and details, thus avoiding hallucination and enhancing credibility. In our survey, our work distinguishes itself from previous studies by defining a paragraph-level granularity, along with the evaluation standard on credibility.
• When designing metrics, we primarily considered Hallucination for factuality, Ambiguity for detailedness, Informativeness for usefulness, and Overall for overall credibility. In our experiment comparisons, RedHat outperformed the baselines up to 2.03%, 5.26%, 6.6%, and 0.137 (Hallu, Ambiguity, Info, Overall), and surpassed SFT by 15.75%, 6.86%, 44.92%, and 0.824, demonstrating the effectiveness of RedHat across these aspects of credibility. Additionally, manual verification showed that the QA accuracy rates were 95.5% for GPT-4 and 85.6% for GLM-4, confirming the effectiveness of the QA mechanism in the process.
• We would like to clarify that human feedback has been fully utilized in our work. The training objective of SFT is to refine human comments. During annotation, annotators were able to compare with the original expert comments. Additionally, we incorporated further annotations on the refined teacher comments, as shown in General Response 1, which we hope addresses your concern.

Dear Reviewer oeU8,

The authors are grateful for your thought-provoking suggestions. Since there are three days left until the discussion period ends, we are unsure whether the responses above has fully addressed your concerns. If there are any remaining issues, please feel free to contact us. We eagerly look forward to your further comments.

Best regards,

The authors

We appreciate your in-depth concerns and criticisms of rooting literature and the human preference part results. They help us reflect on the delivery of our work and improvements.

Connection between induced questions with existing literatures.

To concisely address your concerns, we have reached the Delphi Report [1] (for the purpose of educational assessment), PISA Report [2] (for critical thinking framework), and [3] (for high-school essay assessment in education) in the field of education. We briefly list the criteria from each work. We have sorted the suggested dimensions from the two reports and manually classified the induced questions in Appendix E.1 into the slots, as the following table shows. The induced questions have covered almost all dimensions in acknowledged argumentative evaluation criteria from the perspective of pedagogy and creative-thinking. We hope that such analysis addresses your concerns on rooting literature on argumentation essay evaluation criteria.

One more word, in our manuscript, we have listed our criterion in Appendix C for review and it mainly contains topic, literature, arguments, structure, language and norms. We have also made the classification of induced questions in responses (2/4). In fact, the above examinations are ensured by graduate teaching assistants during experiments.

[1] Facione. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction (The Delphi Report).

[2] Organisation for Economic Co-operation and Development. (2023). PISA 2022 Assessment and Analytical Framework. OECD Publishing.

[3] Noroozi et al. (2023). Design, implementation, and evaluation of an online supported peer feedback module to enhance students’ argumentative essay quality. Education and Information Technologies, 28(10), 12757-12784.

Result convincingness.

As for the follow-up questions on the machenism of methodology, we attribute the success of RedHat due to the better contextual digest ability of ChatGPT-4o than <10 B models and better DocQA accuracy. ChatGPT-4o is better at digest the highlights from the answers, thus contributing to the critique. Also, we adopted the suggestions from Reviewer rhUd and posted the detailed inter-annotator agreement for each score in the main experiment Table. Results show moderate to substantial significance rather than randomness. Please refer to Further discussion on Inter-Annotator Agreement under Reviewer rhUd' comment for details. We further provide the significant-test (T-test) of the results between the base-LLM model and LLM+RedHat results in the following Table. The possibilities for rejection of significance are all lower than 0.05.

Contribution of the paper.

Basically, we kindly remind the reviewer that our core contribution lies in the reduction of hallucination in generated critiques, and results from Table 4 - main experiment - showed reduction in hallucination dimension. While it is a controversial topic whether human preference equals factuality correctness as many recent works in LLM alignment shows, we provide the comparison mainly to indicate that the hallucination reduction might contribute to better preference by human. We respect your suggestion for meticulousness in claiming preference and are willing to make further revisions to the manuscript.

We briefly echo the structure of our validation of RedHat:

1. We have analyzed the similarity between the induced questions and answers in the above response (2/4) and Table 5 in Section 5.3 from the manuscript, showing the answer-similarities are correlated to the reduction in hallucination.
2. Further, the answer accuracy rate for GLM, Qwen, GPT is 85.5%, 92.2%, 95.6% respectively (in Section 5.3 from the manuscript), correlates to the dynamics in the results in Table 4 scores in the major dimensions.
3. We suggest that RedHat helps reduce hallucination in the above mechanism, since RedHat has relieved the core source of hallucination that base-LLM alone is not faithfully grounded on the target essay.

We thank you again for your efforts for reading the lengthy responses. We hope the above materials can solve your concerns and assist you with reevaluating the contributions of RedHat, especially in the aspects of the resources and discussion on the credibility of automatic evaluation for human-written essays - the core motivation of our work.

[2/2] ChatGPT-4o on Artificial Intelligence Conference Paper

To consolidate the validity of our work, we followed [1] and picked a subset of ICLR, Neurips, ACL Conference papers and arxiv papers amounting to 10 with 100 paragraphs to be critiques. We intentionally chose picked those papers whose authors' were accessible so that they could judge the quality over the generated critiques.

Implementation: We applied the ICLR reviewer guidelines as evaluation criteria. Since ICLR reviewer guidelines have already contained more than 10 questions in it, we replace the guideline questions with description of the expectation for a good conference paper on those questions.

Evaluation: to ensure the effectiveness of human annotation, we contacted and invited the authors of the papers to grade along with Hallucination, Ambiguity, Informative, Overall scores along with comparison. The results are shown in the following table. Each critique has received two authors' judgments and the inter annotation agreement reaches 0.74.

Results: results showed an apparent increase in credibility in RedHat assisted critiques. This verified the generalizability of RedHat on English domain writing genres and LLMs.

We will continue to compare with LLaMA-3.1 in English experiments to further validate the effectiveness in English of RedHat, which will be included in the final version of our paper.

[1] Ryan Liu, Nihar B. Shah. ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing. Arxiv. 2023.