Multi-Session Client-Centered Treatment Outcome Evaluation in Psychotherapy

Weakness 1: The TheraPhase dataset is constructed via GPT-4. There is no human evaluation to guarantee the reliability of the dataset.

As described in Section 3.4, the TheraPhase dataset was constructed using GPT-4 by splitting a client’s dialogue data from the CPsyCoun dataset into two phases. This process maintains the integrity of the original data and focuses on creating a structured setup for evaluating treatment outcomes, without altering the underlying assessment or its relevance to treatment progress.

Weakness 2: Experiments only conducted on one or two datasets. It would be better to evaluate on more datasets.

As stated in Line 299, the motivation behind constructing the TheraPhase dataset was precisely due to the lack of existing multi-session datasets for evaluating treatment outcomes in psychological counseling. Currently, there are no publicly available datasets that focus on multi-session client-therapist interactions with sufficient granularity to track changes across sessions.

Weakness 3: I think the proposed approach is not novel. Most parts in the approach are already existing, such as the psychometric test, client profile and PSDI.

While it is true that some components of our framework, such as psychometric tests and client profiles, build upon existing methodologies, our contributions lie in how these elements are integrated and applied to address gaps in evaluating treatment outcomes across multiple sessions.

Weakness 4: In this paper, GPT-4o is selected as the Gold Model. However, as shown in Table 4, its performance is also poor. So there may be deviation in the results.

We respectfully disagree with the assertion that GPT-4o performs poorly. As shown in Table 4, GPT-4o demonstrates the best overall performance compared to other models, making it the most suitable choice as the Gold Model for our evaluation framework.

Confused by the figure 2: the PSDI<0 should mean the good treatment outcomes but the sad face there indicates negative results. Can you clarify this?

In our framework, treatment outcomes are determined by the change in PSDI values across sessions, rather than the absolute value at any specific point. A negative PSDI change (PSDI < 0) represents a decrease in symptom severity, indicating positive treatment outcomes, even if the face in Figure 2 might suggest otherwise. We acknowledge that this visual representation could be misleading and will revise Figure 2 to better align the iconography with the intended meaning, ensuring it reflects the relationship between PSDI change and treatment outcomes as described in Section 3.

Some useful but undiscussed related works are [1], [2] and [3]

We thank the reviewer for pointing out these related works, which offer valuable perspectives in the field. Below, we summarize each study and its relevance to our work:

• [1] This study proposes a human-machine collaboration framework, Roleplay-doh, which enables domain experts to create LLM-simulated patients for novice counselors to practice with. While innovative, this work primarily focuses on training tools for novice counselors and does not align with the goals of our study, which centers on treatment outcome evaluation.
• [2] This work introduces the PATIENT-Ψ framework, leveraging LLMs to simulate patient behaviors for mental health professionals’ training. Similar to Roleplay-doh, this study focuses on creating realistic patient simulations for training purposes, which differs fundamentally from our focus on evaluating treatment progress and outcomes.
• [3] This study evaluates the cross-lingual performance of LLMs in healthcare queries and highlights significant performance gaps in non-English settings. This is particularly relevant to the discussion of language's influence on task performance, as noted in Section 4.1, and we appreciate the reviewer’s suggestion to incorporate this perspective into our discussion.

While we find the first two works less relevant to our research focus, we agree that incorporating more discussions on psychological and healthcare-related studies can benefit the community’s understanding of this task. We will include a broader discussion of these works and their implications in the appendix to provide additional context for interested readers.

[1] Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles (EMNLP 2024)

[2] PATIENT-𝜓: Using Large Language Models to Simulate Patients for Training Mental Health Professionals (EMNLP 2024)

[3] Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries (WWW 2024)

Weakness 3: The goal and contributions of the paper are unclear. The IPAEval is targeted to evaluate the treatment outcomes (by authors’ definition is the clients’ states etc) but the authors also evaluate psychological assessment (e.g. symptom detection and severity assessment).

First, we would like to address a misunderstanding: in our paper, we did not define treatment outcomes solely as clients’ states. As illustrated in Figure 1: "What is Treatment Outcome?" and explained in Line 146, treatment outcomes are conceptualized as the changes observed in psychological assessments across different stages of the treatment process. While psychological assessments provide a snapshot of the client’s mental state, treatment outcomes are derived from tracking how these assessments evolve over time.

To this end, our experiments are structured into two parts. First, we evaluate psychological assessments (e.g., symptom detection and severity assessment) as foundational tasks, providing the necessary data to establish the client’s mental state at specific stages. Second, we evaluate treatment outcomes by analyzing the changes in these assessments across sessions, enabling us to model the dynamics of client progress over time.

Weakness 4: The “item-aware reasoning” sounds misleading as it does not contain a multi-step thinking process on rationales but only one intermediate step of giving the rationale behind the item classification. Maybe consider changing the naming of reasoning or provide more concrete examples/analysis.

The term “item” in our work originates from psychometric questionnaires used in psychological assessments, where "items" refer to individual questions or prompts designed to measure specific psychological constructs. To clarify, in our framework, “item-aware reasoning” specifically refers to the process of generating rationales tied to the classification of each item within the psychological assessment, rather than multi-step reasoning on broader rationales. If the reviewer could provide more details on the source or specific reference for the interpretation of “item-aware” or “multi-step thinking process on rationales,” it would help us to include a discussion in the appendix, distinguishing our terminology from the alternative usage.

Question 1: I’m confused by several terms – psychological assessment, symptom detection, severity assessment. It sounds to me that symptom detection, severity assessment are parts of psychological assessment. Can you clarify these definitions better?

Your understanding is correct: symptom detection and severity assessment are indeed components of psychological assessment. Specifically:

• Psychological Assessment refers to the overall process of evaluating an individual’s mental state using standardized tools, questionnaires, or structured interviews. It encompasses various methods and measures designed to assess psychological constructs.
• Symptom Detection focuses on identifying the presence or absence of specific psychological symptoms.
• Severity Assessment evaluates the intensity or degree of the identified symptoms to understand their impact on the individual’s functioning.

We acknowledge that our current manuscript lacks sufficient explanation of these terms and their relationships, which may have led to confusion. To address this, we will include a dedicated Background Section in the revised manuscript, providing clear definitions and context for key psychological concepts and how they are used in our framework.

Question 2: I noticed that there are contrasting results between models on symptom severity in Figure 4 – any reason/hypothesis to explain this?

We would like to clarify that there is no inconsistency between the models’ performance on symptom detection and symptom severity assessment. The results in Figure 4 are part of an ablation study aimed at understanding the impact of item-aware reasoning across different tasks.

The first subfigure corresponds to the symptom detection task, where performance is measured using metrics such as F1 score (higher is better). The second subfigure corresponds to symptom severity assessment, where performance is evaluated based on error metrics (lower is better), as detailed in Section 4.1. The difference in the directionality of these metrics (higher for F1, lower for error) might give the impression of contrasting results, but this is due to the nature of the evaluation metrics used for each task rather than inconsistencies in the findings.

Thank you for your thoughtful and constructive review, which provides valuable insights into the strengths and areas for improvement in our work. We appreciate the detailed feedback and are grateful for the opportunity to address the points raised.

Weakness 1: The main claim of the paper is to evaluate the treatment outcomes across sessions. However, there is no validation/exploration on the TheraPhase dataset to show that it is reflective of real world outcomes (i.e. to ensure the distributions of treatment outcomes across time is close to real-world clients).

We appreciate the reviewer’s concern regarding the representativeness of the TheraPhase dataset in reflecting real-world treatment outcomes. The TheraPhase dataset is built on CPsyCoun [1], a rigorously evaluated and widely recognized dataset in psychological counseling research. CPsyCoun has been validated across multiple psychotherapy schools, including psychodynamic, cognitive-behavioral, and integrative approaches, demonstrating its ability to capture client-therapist interactions and treatment dynamics in diverse therapeutic contexts.

The original CPsyCoun study showed significant improvements in professional metrics, such as interpretability, adaptability to different therapeutic frameworks, and relevance to real-world counseling practices. These validations by domain experts underscore CPsyCoun’s reliability and representativeness as a foundational dataset.

By leveraging CPsyCoun, TheraPhase maintains its grounding in these professionally validated practices while focusing on session-specific treatment outcome evaluations. This strong foundation ensures its relevance and applicability to real-world scenarios, even as we continue to explore broader validations in future work.

[1] CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling (ACL 2024)

Weakness 2: The TheraPhase dataset is based on a Chinese counseling dialogue dataset. The performance difference between models (e.g. Llama vs. Qwen) may be affected by the language's comprehension ability, rather than the tracking ability on treatment outcomes. The findings on this dataset is questionable.

We thank the reviewer for raising this important concern regarding the potential influence of language comprehension ability on model performance. This is precisely why we conducted experiments using a diverse range of models, encompassing both open-source and closed-source LLMs across various parameter sizes. The closed-source models tested include GPT-4, GPT-4-turbo, GPT-4o, and GPT-4o-mini, representing OpenAI's latest advancements. Additionally, we evaluated state-of-the-art open-source models, such as Llama3.1-40B, Llama3.1-70B, Qwen2-72B, Mistral-8X22B, and Mistral-8X7B. This comprehensive set of models ensures a robust evaluation under consistent experimental conditions, mitigating potential biases introduced by language comprehension differences.

We acknowledge the absence of an in-depth analysis of language-specific effects on treatment outcome tracking. In the course of our study, we considered translating the original Chinese data into English to investigate the influence of language. However, a recent study [2] has highlighted that depression detection models trained in English exhibit significant misjudgments when applied to data from other cultural backgrounds. To avoid introducing cultural biases into our experiments, we decided against translating the original Chinese data into English. The field currently lacks multi-session, multilingual datasets focused on therapy outcomes, and developing such datasets is beyond the scope of this study. Our primary goal is to evaluate models’ ability to handle multi-session treatment data within a well-defined and representative context, leveraging the existing resources effectively.

[2] Diverse Perspectives, Divergent Models: Cross-Cultural Evaluation of Depression Detection on Twitter (NAACL 2024)

Thank you for your detailed responses, which addressed part of my concerns. I am happy to raise my rating to 5. However, I still believe that verifying the "multi-session" aspect (to determine if it is close to real world) is crucial. This verification would significantly strengthen the contribution of this work and better differentiate it from other studies.

First, we evaluate psychological assessments (e.g., symptom detection and severity assessment) as foundational tasks, providing the necessary data to establish the client’s mental state at specific stages. Second, we evaluate treatment outcomes by analyzing the changes in these assessments across sessions, enabling us to model the dynamics of client progress over time.

Thanks for explaining this. I recommended authors can frame the paper to show a more clear flow based on this in the future revision.

“item-aware” or “multi-step thinking process on rationales,” it would help us to include a discussion in the appendix, distinguishing our terminology from the alternative usage.

I understand the "item-aware" terminology but I worry if a reader less familiar with psychotherapy may find it confusing. Authors may consider explicitly specifying this as e.g. "psychometric-items"

For the "reasoning", I am not suggesting that this term is an incorrect description of the authors' rationale approach. I think the word "reasoning" has recently become overloaded with various additional meanings (e.g. the popular o1 model). Readers may expect multi-hop reasoning process. I think it is very minor point, and authors can consider how they want readers to interpret it.

no inconsistency between the models’ performance on symptom detection and symptom severity assessment

I see! Thanks for clarifying this (F1 vs Mean absolute error). I recommended to add a upper arrow/ down arrow to indicate that the y-axis are different.

For the figure 4, another question I am not sure:

apart from that, the (arguably stat. insignificant) results (Figure 4: error bars overlapping before and after the reasoning prompt) in tracking therapeutic outcomes suggests a deeper analysis on the therapeutic outcomes e.g. which part of the therapeutic outcomes can be tracked better.

--> do authors have any explanation or hypothesis on this?

Question 1: Could you provide more detail on why only 30 sessions were annotated for psychological assessment and 60 for treatment outcomes, given that these numbers are quite low relative to the available dataset? While I understand that annotating the entire dataset may not be feasible, expanding beyond this limited sample would likely improve the robustness of IPAEval’s evaluation.

We thank the reviewer for raising this point. The decision to annotate only 30 sessions for psychological assessment and 60 sessions for treatment outcomes stems primarily from the significant time and resource requirements for high-quality annotations. Each session requires detailed labeling, with 10 annotation points per session across four categories, resulting in a total of 900 annotations. This process involves extensive manual effort from domain experts, making it challenging to scale up further within the scope of this work.

While we agree that annotating a larger sample could improve the robustness of the evaluation, we designed our approach to maximize the utility of the available annotations. Rather than training the model directly on this limited dataset, we use it solely for evaluation purposes—to select the best-performing models for broader testing. This strategy minimizes potential biases while ensuring the dataset is used effectively.

Question 2: Additionally, have you considered using multiple LLMs to cross-annotate the dataset? Cross-annotation with multiple models could help improve labeling quality and consistency while reducing the manual effort required, potentially increasing the reliability of the labeled data.

In the early stages of our study, we indeed considered this approach, as it offers significant cost savings and scalability. However, given the sensitivity of mental health-related data, even annotations generated by multiple LLMs would require thorough review and validation by mental health professionals to ensure their reliability. This additional validation process offsets some of the cost savings and introduces complexity in scaling the annotation pipeline.

Furthermore, the field currently lacks high-quality, publicly available mental health datasets, which limits the utility of automated cross-annotation without a strong reference framework. While fully manual annotation is costly and narrow in scope, it ensures the production of a gold-standard dataset that can serve as a robust foundation for future research in the community. We believe such datasets, though smaller in size, are critical for advancing research in this domain, where accuracy and trustworthiness are paramount.

Thank you for your thoughtful and constructive review, which provides valuable insights into the strengths and areas for improvement in our work. We appreciate the detailed feedback and are grateful for the opportunity to address the points raised.

Weakness 1: Despite having 110 and 800 client sessions available, the authors only annotated 30 for psychological assessment and 60 for treatment outcomes, using this small subset to establish a “Gold Model” for generating reference scores. This approach may introduce bias, as basing the Gold Model on such a small sample could affect the reliability of the labeled data, especially in representing the full diversity of client interactions.

We appreciate the reviewer’s concern regarding the size of the labeled dataset used to establish the "Gold Model." The limited number of annotated sessions is due to the high cost and effort involved in labeling each session, as each sample requires 10 annotation points across four categories, resulting in a total of 900 annotations. While the absolute number of sessions may appear small, the total annotation volume is substantial and provides a robust basis for evaluation.

Regarding the potential introduction of bias, we would like to clarify that the annotated data is not used to train the model. Instead, it serves as an evaluation mechanism to identify relatively better-performing models, which are then used to evaluate other models more broadly. We acknowledge that the limited size of the labeled dataset may not fully represent the diversity of client interactions; however, it is sufficient to differentiate models on this specific task. Furthermore, this approach avoids introducing training bias and instead highlights discrepancies between human annotations and model predictions, as well as differences among various models.

Additionally, we would like to point out that many other well-established datasets [1] [2] in the mental health domain feature data volumes of a similar scale, typically around 1,000 annotated samples. This demonstrates that our dataset aligns with the standard practices in the field, further supporting its adequacy for the intended evaluation purposes.

[1] Supervised Learning and Large Language Model Benchmarks on Mental Health Datasets: Cognitive Distortions and Suicidal Risks in Chinese Social Media

[2] Identifying Depression on Reddit: The Effect of Training Data

Weakness 2: The paper presents IPAEval as an improvement over models like ClientCAST and CPsyCoun, emphasizing its client-centered, multi-session approach. However, it doesn’t provide direct performance comparisons to back up these claims. While Table 1 outlines conceptual differences (such as focus and theoretical grounding), there’s no side-by-side benchmarking on tasks like symptom detection or outcome prediction. Without this empirical evidence, IPAEval’s advantages remain mostly theoretical. Adding direct comparisons with existing models would make the practical impact of IPAEval clearer and more convincing.

We would like to clarify that these methods are fundamentally designed to evaluate therapists' performance rather than focusing on tasks such as symptom detection or outcome prediction. In contrast, IPAEval is client-centered, aiming to assess client outcomes and interactions over multiple sessions. These differing objectives make direct benchmarking less relevant or meaningful, as the core tasks and evaluation criteria for these methods diverge significantly.

Furthermore, while Table 1 highlights the conceptual distinctions between IPAEval and existing models, our focus is not on competing directly with therapist-evaluation methods but rather on addressing a gap in client-centered evaluation frameworks. We agree that demonstrating IPAEval's practical utility is essential, and we have prioritized task-specific evaluations to ensure its relevance to client-focused outcomes.

Weakness 3: Figure 2, which aims to show the whole IPAEval framework, doesn’t clearly illustrate the Session-Focused Client-Dynamics Assessment module, especially how it tracks changes within a session from start to end. This mismatch between the figure and the text can make it confusing for readers trying to understand how session-specific changes are actually captured. Adding markers for initial and final assessments within each session would make the figure align better with the explanation in the text and help clarify how this part of the framework works.

We acknowledge that the current figure does not adequately illustrate how session-specific changes, particularly from start to end, are captured. This could indeed cause confusion for readers attempting to fully grasp this part of the framework. To address this, we will revise Figure 2 in the final version by adding explicit markers for initial and final assessments within each session.