MoodAngels: A Retrieval-augmented Multi-agent Framework for Psychiatry Diagnosis

We really acknowledge your thoughtful questions, and we revised our paper to clarify them and add experiments on the ablation study and analysis.

W1/Q1 Clinical Significance of Incremental Performance Gains While the improvement from Angel.R to the multi-agent system may appear modest in statistical terms, even marginal gains in diagnostic accuracy hold substantial clinical value. In psychiatric practice, where each case carries significant implications for patient care and treatment outcomes, these incremental advancements can translate into meaningful benefits for a broader population of individuals seeking mental health support.

W2/Q2 Statistical Significance Test Reults

• Interpretation of p-values: All models except llama3-8b-instruct on real cases have p-values far less than 0.05, indicating that their predictive performance is significantly better than random guessing.

• 95% Confidence intervals (CI): The 95% confidence intervals for each model are relatively narrow, suggesting that the results have high statistical reliability.

Results of real cases:

Results of synthetic cases:

W3. Ablation Study on Retrieval Components The strong performance of majority voting validates the diagnostic capability of individual agent judgments. However, the debate process remains essential for ensuring comprehensive symptom analysis through structured deliberation, maintaining output stability via evidence-based adjudication, and providing interpretable reasoning traces for clinical validation.

W4. Top-5% Scale-Item Threshold Selection In designing our threshold criteria, we carefully considered this important aspect. Through a review of established psychiatric methodologies [1], we observed that prior work typically derives thresholds through a combination of clinical expertise and empirical data distribution analysis. To ensure clinical validity, we first computed Pearson correlation coefficients for each scale item, then engaged in extensive discussions with our clinician co-authors to determine appropriate thresholds. Notably, based on their expert recommendations, we incorporated additional items (PHQ-9 Q1 and Q2) that demonstrated particular clinical relevance, as detailed in lines 697-700 of our manuscript.

[1]Unraveling the distinction between depression and anxiety: A machine learning exploration of causal relationships.

W4/Q3. Retrieval Top-k Similar Cases Threshold Selection We will include an ablation study that varies the top-k value (0, 1, 3, 5) for case retrieval, and we will report the results in the rebuttal once testing is complete. Notably, we observed that Angel.R already performs well even without retrieval (k=0), which further supports the robustness and quality of our overall framework.

Q4. Choice of Retriever We chose BGE-M3 for its strong semantic representation capabilities, which align well with our goal of capturing nuanced symptom similarities in clinical texts. While we agree that exploring alternative retrievers could offer additional insights, our current focus has been on validating the overall effectiveness of the framework. Notably, as shown in W3, even in the absence of any retrieval component (Angel.R w/o DSM-5 Symptom Matching), the model still achieves promising results, suggesting the robustness of the reasoning process. We hope this addresses your concern, and we truly value your encouraging feedback.

Q5. Judge's Criteria for Ending the Debate. We designed the prompt of the Judge Agent to simulate a debate format, with a minimum of 2 rounds and a maximum of 4. At the beginning of each round, the prompt guides the Pro and Con agents through a structured process: Round 1 for opening statements, Round 2 for rebuttals, Round 3 for clarifying unresolved points, and the final round for closing arguments. After each round, the Judge Agent evaluates whether the debate should continue, provides reasoning, and reflects on the arguments. At the end of the debate, the judge delivers a diagnosis with reasons. 

Q6. Quality of Knowledge Base. The diagnostic and differential criteria from the DSM-5 were originally presented as bullet points, as shown in Appendix C1. We used GPT-4o to revise these items into concise, self-contained statements, rather than generating new diagnostic content. This step was necessary because some original entries contained pronouns or overly detailed explanations that were not helpful for symptom matching between the knowledge base and medical records. After constructing the knowledge base, each extracted item was reviewed and verified by our domain expert.

Q7. Distribution of real cases We are calculating the detailed types, and we will report the result in this rebuttal.

Q8. Medical Record Structuring We designed medical record structuring as an optional component to ensure the flexibility of our framework, particularly in scenarios where medical records lack clarity or where input token limitations need to be considered.

Q9. Distinguishing Mood Disorders. Currently, our framework performs binary classification—determining whether a mood disorder is present. While we have extracted the DSM-5 differential criteria for more fine-grained distinctions, we do not attempt subcategory classification. This is because psychiatric diagnosis requires careful deliberation; in clinical practice, clinicians typically first identify a broad diagnostic direction before narrowing it down through further assessments. Similarly, our framework is designed to support this initial diagnostic stage.

Q10. Terminology for Individuals Seeking Help Thanks for raising this important point. We also place great importance on using professionally appropriate terminology. For individuals who have not been formally diagnosed with a mental disorder, we use terms such as client or visitor to show respect and avoid assumptions. Once a diagnosis is confirmed, we refer to them more specifically as patients. For example, as in line 276, where we state “a patient with severe depression.”

L1. Usage Declaration of the Diagnostic Framework Thanks. We will revise our paper and GitHub repo to add a usage declaration.

Dear Reviewer,

The author has provided a rebuttal to respond to your comments. Please have a read on the author response and discuss with author if necessary.

Thanks,

AC Team

I appreciate the new ablation studies and clarifications to my questions. Your rebuttal have strengthened the paper (please add them into the main context or appendix if possible) and addressed most of my initial concerns.

Note that your response has also raised a point for discussion that shows removing DSM-5 symptom matching or replacing the debate with a simple majority vote leads to a very minimal drop in performance. This reinforces my original concern (Q1): the quantitative evidence does not quite support the necessity of the added complexity of the full multi-agent debate system over the simpler, high-performing Angel.R or a simple majority vote. Also, while I respect your argument about the clinical value of marginal gains, the paper would be more balanced if it explicitly acknowledged that the empirical performance boost from the debate is not very significant. You may highlight, as you do in the rebuttal, that its value lies in other areas like interpretability or handling edge cases or something else. This transparency would make the manuscript more robust. Finally, your response for Q9 is also a detail that helps in scoping the contributions and limitations. I recommend making this explicit in the paper.

I am raising my score from 4 to 5. The authors provided a helpful rebuttal that addressed nearly all of my concerns with ablation experiments, statistical analyses, and detailed clarifications.

Why not higher/lower see final justification section.

W1. Clarification on Methodological Innovations We appreciate the reviewer’s feedback and would like to clarify that our framework is not merely a combination of existing components, but introduces several key innovations tailored to the unique challenges of psychiatric diagnosis:

• Granular Scale Decomposition: Instead of treating scales as fixed global scores, we restructure them into item-level diagnostic signals. This mitigates the limitations of self-reported or clinician-rated scale scores by enabling finer-grained and more interpretable analysis.
• Retrieval-Augmented Symptom Matching: Unlike generic RAG applications, we apply retrieval specifically to address the symptom overlap problem in psychiatric diagnosis, leveraging structured DSM-5 criteria and expert-annotated examples to improve diagnostic precision in ambiguous cases.
• Debate-Inspired Multi-Agent Reasoning: We introduce a novel debate mechanism, inspired by formal debate tournaments, in which independent diagnostic agents present their reasoning and a non-intervening judge agent synthesizes the outcome. This design enables transparent multi-perspective evaluation and handles difficult or conflicting cases with human-like deliberation.

Together, these innovations go beyond simple component reuse and are purpose-built for the nuanced reasoning required in mental disorder diagnosis.

W2/Q1. Medical LLM as backbone We are testing the results of medical llm as backbone, and we will report the result in this rebuttal as soon as we finish the experiment.

Q2. Synthetic data generation methods We appreciate the insightful suggestion regarding synthetic data. While our primary contribution lies in the diagnostic framework, the synthetic dataset was developed as an additional effort to address the challenges of data privacy and the lack of high-quality public datasets in psychiatric diagnosis. Currently, we employ an off-the-shelf method for case generation, and we are actively exploring ways to incorporate more diverse datasets and multimodal inputs to further advance AI-driven psychiatric diagnosis.

Dear Reviewer,

The author has provided a rebuttal to respond to your comments. Please have a read on the author response and discuss with author if necessary.

Thanks,

AC Team

W1.1 Clinical validation of diagnostic output We evaluated their performances by three licensed psychiatry experts on our co-author clinician’s team. Metrics are clinical coherence (0-2 points), information completeness (0-2 points), and clarity of expression (0-2 points).

Clinical Coherence: whether reasoning aligns with clinical logic and evidence.

• 2 = Fully coherent and professionally sound
• 1 = Minor logical flaws
• 0 = Major errors or conflicts

Information Completeness: inclusion of key diagnostic elements (e.g., symptoms, scales, course, exclusions, impairment).

• 2 = Comprehensive and sufficient
• 1 = Missing important elements
• 0 = Lacks key diagnostic info

Clarity and Expression: clarity and professionalism of the explanation.

• 2 = Clear and professionally worded
• 1 = Some ambiguity or terminology misuse
• 0 = Unclear or confusing

Here are the assessment results:

We report Fleiss' Kappa as the measure of inter-annotator agreement (IAA).

The results demonstrate the high quality of our generated outputs. The higher inter-annotator agreement suggests that the outputs of our multi-agent system are more consistent, clinically aligned, and better recognized by experts.

W1.2 Literature Basis for Item-Level Analysis Traditional psychiatric diagnostic approaches have predominantly relied on total score interpretations of assessment scales, potentially obscuring crucial symptom-level heterogeneity. Recent advances in depression research have highlighted the importance of granular-scale analysis to overcome this limitation. [1] demonstrated how item-level variability in depression scales (e.g., distinguishing somatic vs. cognitive symptoms) challenges conventional unidimensional scoring. Beyond this, [2] identified core depressive symptoms through factor analysis of individual Hamilton Scale items, while van [3] conclued clinically meaningful MDD subtypes using PHQ-9 item clustering. Through close collaboration with clinical experts, our MoodAngels framework extends this paradigm by systematically grouping scale items to reduce assessment bias and enhance diagnostic precision.

[1] Measuring depression over time… Or not? Lack of unidimensionality and longitudinal measurement invariance in four common rating scales of depression. In Psychological Assessment, 28(11), 1354–1367.

[2] An Item Response analysis of the Hamilton Depression Rating Scale using shared data from two pharmaceutical companies. In Journal of Psychiatric Research, 38(3), 275-284.

[3] Data-driven subtypes of major depressive disorder: a systematic review. In BMC medicine, 10(1), 156.

W1.3 Dependence on Clinical Interviews Our framework incorporates medical records and five clinician-administered scales, all of which rely on structured clinical interviews. Rather than replacing clinical interactions, our system is designed to assist clinicians after the full interview process is completed, by supporting diagnostic reasoning based on the comprehensive set of information gathered from the patient.

W2.1/W2.3/W2.4 Human Evaluation of Synthetic Dataset Our synthetic dataset consists of scale-based numerical data. We initially filtered the data by ensuring the sum of subscores did not exceed the total score. Additionally, we evaluated how closely the synthetic data resembled real data using automated metrics (see Line 877, achieving a score of 0.87). For human evaluation, we randomly sampled 50 cases (25 real and 25 synthetic) and asked three licensed clinicians to distinguish between them. Their classification accuracy was as follows:

W2.2 Pattern Accuracy of Synthetic Data In line 847, we evaluated Pattern Accuracy (α=0.72), which measures how precisely the synthetic data reproduces the core statistical patterns of real data, ensuring generated samples are realistic and representative.

W2.4 Comparisons to existing psychiatry datasets Current scale data are mainly one self-reported scale for the depression detection task. We found no existing dataset enough for mood disorder diagnosis. Therefore, besides our main contribution of designing a diagnostic framework and testing its performance on real clinical data, we constructed a synthetic dataset to prompt academic research in the field.

W3. Medical baselines We tested the result of medgemma-27b-text-it, which is an advanced LLM provided by Google. We found that in real cases, it tends to diagnose nearly every case with mood-disorder-related symptoms in the medical record as mood disorder, without having the sensitivity of overlapping symptoms in other mental disorders.

Result on synthetic data: recall: 0.67, acc: 0.743, mcc: 0.57, macro f1: 0.717.

Result on real data: recall: 0.535, acc: 0.704, mcc: 0.525, macro f1: 0.704.

Dear Reviewer,

The author has provided a rebuttal to respond to your comments. Please have a read on the author response and discuss with author if necessary.

Thanks,

AC Team

We are grateful for your thoughtful questions, and we have revised our paper to address them and presented examples of our framework's output.

Q1. A debate case, where only Angel C was correct and the other two were wrong, and the Judge made a correct decision by debate.

[Angel R's final output] The client has a mood disorder.

• The client shows mood instability with shifts from depression to (hypo)mania, consistent with bipolar disorder.
• Past hospitalization for mood disorder and long-term use of mood stabilizers support this.
• Symptoms like irritability, increased activity, and energy align with hypomanic episodes per DSM-5.

[Angel D's final output] The client has a mood disorder.

• The client exhibits emotional instability, including hallucinations and inappropriate laughter.
• Despite clinician-rated scales showing no depression, self-reports suggest moderate depression.
• Similar cases were diagnosed as bipolar or major depression, reinforcing this possibility.

[Angel C's final output] The client doesn't have a mood disorder.

• Though the client self-reports mild depressive and anxiety symptoms, clinician ratings show no signs of depression, anxiety, or insomnia.
• Mood fluctuation history is noted, but current assessments do not support an active mood disorder diagnosis.
• For hallucinations and inappropriate laughter, consider the client having schizophrenia.

---

[Judge] Round 1 begins. Both sides should claim the diagnosis and reasons in this round.

[Positive] I believe this client has a mood disorder.

• The client's records show long-term mood fluctuations, ranging from depressive to elevated states, which is characteristic of bipolar disorder.
• The client was previously hospitalized for a mood disorder and has been on mood-related medication, indicating a strong link to mood disorder symptoms.
• Although clinician-rated assessments did not indicate significant depression or anxiety, the client's self-report showed moderate depression, which may better reflect their subjective emotional state.

[Negative] I don't think this client has a mood disorder.

• Clinician-rated assessments show no signs of depression, anxiety, or suicidal tendencies—key indicators of mood disorders.
• Although the client initially improved with mood-related medication, new symptoms appeared despite continued medication use, suggesting a possible different mental condition.
• Similar cases have inconsistent diagnoses, so relying solely on past hospitalization records is insufficient for a conclusive diagnosis.

[Judge] Rebuttals missing. Continue.

---

[Judge] Round 2 begins, both sides should refute the opposite side.

[Positive]

• Self-reported symptoms indicate depression and suicidality, with a PHQ-9 score of 19 and daily suicidal ideation. These are core indicators of mood disorders, even if clinician-rated scales show minimal symptoms.
• New symptoms during continuous medication do not rule out mood disorders. Mood disorders often present episodically or with partial response to treatment.
• Historical diagnosis and treatment response support a mood disorder. The client was previously hospitalized for an affective disorder and showed improvement with mood stabilizers.

[Negative]

• Mood fluctuations alone do not confirm a mood disorder. The observed shifts could be part of other psychiatric conditions such as borderline personality traits or stress-induced responses. Symptom variability requires a broader differential diagnosis.
• Past hospitalization and medication use do not confirm the current diagnosis. Historical treatment for a mood-related condition is not definitive evidence of a present mood disorder, especially when recent clinician-rated assessments show no active depressive or anxious symptoms.
• Self-report scores must be interpreted cautiously. While PHQ-9 indicates moderate depression, clinician-administered scales (HAMD, HAMA, BPRS) consistently show no depressive, anxious, or suicidal signs. Clinical observation remains the gold standard.

[Judge] Certain points are not discussed. Continue.

---

[Judge] Round 3. Please discuss the possibility of a performance conflict between self-report and clinician assessment, in one sentence.

[Positive] The client's symptoms are suppressed by medicine.

[Negative] The client may have schizophrenia.

[Judge] All symptoms discussed. Debate ends. The client doesn't have a mood disorder. Although the client was previously hospitalized and prescribed medication related to mood disorders, multiple clinician-administered scales did not indicate typical symptoms of depression or anxiety. Moreover, the client's current symptoms are not alleviated by mood disorder medications, raising the possibility of other psychiatric conditions with overlapping features, such as schizophrenia.

Q2. We designed the prompt of the Judge Agent to simulate a debate format, with a minimum of 2 rounds and a maximum of 4. At the beginning of each round, the prompt guides the Pro and Con agents through a structured process: Round 1 for opening statements, Round 2 for rebuttals, Round 3 for clarifying unresolved points, and the final round for closing arguments. After each round, the Judge Agent evaluates whether the debate should continue, provides reasoning, and reflects on the arguments. At the end of the debate, the judge delivers a diagnosis with reasons. The example in Q1 demonstrates a case where the debate concluded in three rounds.

Q3. Thanks for your suggestion. We are testing on changing the temperature of the angel and the different prompt order of the debate part. We will display the result as soon as we complete the experiment.

Q4. The debate agents generate reasons for their position by points. We evaluated their performances by three licensed psychiatry experts on our co-author clinician’s team. Metrics are clinical coherence (0-2 points), information completeness (0-2 points), and clarity of expression (0-2 points).

Clinical Coherence: whether reasoning aligns with clinical logic and evidence.

• 2 = Fully coherent and professionally sound
• 1 = Minor logical flaws
• 0 = Major errors or conflicts

Information Completeness: inclusion of key diagnostic elements (e.g., symptoms, scales, course, exclusions, impairment).

• 2 = Comprehensive and sufficient
• 1 = Missing important elements
• 0 = Lacks key diagnostic info

Clarity and Expression: clarity and professionalism of the explanation.

• 2 = Clear and professionally worded
• 1 = Some ambiguity or terminology misuse
• 0 = Unclear or confusing

Here are the assessment results:

We report Fleiss' Kappa as the measure of inter-annotator agreement (IAA).

The results demonstrate the high quality of our generated outputs. The higher inter-annotator agreement suggests that the outputs of our multi-agent system are more consistent, clinically aligned, and better recognized by experts.

Q5. Thank you sincerely for the thoughtful suggestion. Our judge is entirely language-model-based and is designed to simulate a debate adjudicator rather than proposing new diagnostic hypotheses. The DSM-5 alignment is primarily handled in earlier stages by the three independent diagnostic agents, whose outputs serve as the “debate material.” In this setting, the Judge is more like a referee than a debater, who does not gather evidence or match symptoms directly to criteria.

Dear Reviewer,

The author has provided a rebuttal to respond to your comments. Please have a read on the author response and discuss with author if necessary.

Thanks,

AC Team