DynaEval: A Dynamic Interaction-based Evaluation Framework for Assessing LLMs in Real-world Scenarios

We would like to thank you very much for the detailed feedback and valuable suggestions.

The “ChatGPT” in experiments means GPT-3.5-turbo. Sorry for the confusion. We will clarify this in the new version of our paper.

W1.a. Even though EGPI seems to be a central concept in their paper, the authors do not even explain what it is. They simply brush over it by adding a citation.

A1. a. Thank you for your question. We explained EGPI in our paper in p3 (after the citation), and introduced the formal definition of EGPI in Appendix A.2. Due to the page limitation, we did not place the definition and detailed explanation of EGPI in the main body of this paper.

W1.b. The DynaEval interaction process doesn't seem quite natural and it's not explained from where the "interaction goal" and "interaction rule", which are supposed to be defining characteristics of these processes, come from.

A1.b. As mentioned in the first paragraph of Section 2.2, it should be noticed that the definition of dynamic interaction is totally based on our observation of real-world dynamic interaction rather than matching with EGPI. The relationship between DynaEval and EGPI is an accidental discovery, and we find that EGPI can help us to standardize the evaluation of LLMs. Indeed, the “interaction goal” and “interaction rule” are common denominators of dynamic interactions and are sufficient to depict the former. For example, the review-rebuttal process of ICLR is a dynamic interaction scenario, which can be completely depicted by interaction goals (of reviewers and authors) and interaction rules. For reviewers, their interaction goal is to accept high-quality papers and reject low-quality papers and to help improve the quality of papers. For authors, their interaction goal is to explain to and discuss with reviewers to maximize the probability of publishing their paper at this conference and also improve the quality of their paper. The interaction rule of the review-rebuttal process is that reviewers first give their original reviews of submitted papers, and then reviewers and authors can freely discuss the paper during the rebuttal phase.

W1.c. The equations (1) and (2) seem totally out of place in the paper. Honestly, I'm not sure what role they play in explaining any of the concepts. Are they just there to explain the concept of sampling, or are they there simply to provide a "scare factor"?

A1.c. Thank you for your doubt. The equations (1) and (2) aim to model and alleviate the uncertainty of LLMs’ performance in dynamic interaction-based evaluation, which is significant in LLM evaluation. As shown in our experiments, the performance of LLMs is unstable and can fluctuate given different interaction environments. Unlike traditional supervised signal-based evaluations where the environment (context) is static, the environment of dynamic interaction-based evaluation is itself dynamic, thus magnifying the uncertainty of LLMs’ performance. Therefore, it is indispensable to formally model the performance of LLMs with “uncertainty” and to alleviate the effect of uncertainty on evaluation results. As a result, rather than modeling the performance of LLMs by deterministic values, we find that modeling by random variables is more suitable for this problem and utilize equations (1) and (2) to present how our method models and alleviates the uncertainty of LLMs’ performance in the evaluation process.

W2.a. Both Figures 3.a and 3.b show a set of highly overlapping distributions. The medians of the top contenders in 3.a are barely distinguishable even, however the conclusion that section 3.2 notes is that GPT-4 performs best is mode 1. It seems to me like the authors are making claims without regard to statistical significance of their results.

A2.a. Thank you for your constructive suggestion. Statistical significance can be used to test whether specific features of random variables (e.g., expectation) are different. In our paper, statistical significance is not used for two reasons. First, one indispensable assumption of most hypothesis testing methods (e.g., t-test and chi-square test) is that random variables should follow a family of distributions like normal distribution. Although this assumption is valid in many real-world scenarios (e.g., a person’s height), it is unconfirmed for the random variable of an LLM’s performance, and there is still a lack of research on the statistical property of LLMs’ performance. As a result, we cannot ensure whether the statistical significance itself is correct. Second, rather than to “rank the ability of different LLMs”, we aim to use DynaEval to “discover and depict the ability of different LLMs using evaluation”. Therefore, we presented the difference in LLMs’ performances in terms of average score and stability (the range of the box) in the text of Section 3.2. Following your constructive suggestion, we will discuss more about the performances of LLMs in the next version of our paper.

W2.b. The same is true in Table 2: the difference between GPT-4 (8.77) and ChatGPT (8.74) seems negligible, but that is not reflected in the conclusions of the text.

A2.b. Thank you for your constructive suggestion. As explained in A2.a, we will supplement more discussion about performance of LLMs in Code G&R (especially Table 2) in the next version of our paper.

W3.a. I'm not convinced that DynaEval framework itself brings anything to the table.

A3.a. Thank you for your doubt. As you mentioned, using games as evaluation methods depends on the implementation. One challenge in this process is that games can differ a lot in many aspects such as the number of participants, role design, game rule, etc., and are hard to standardize. As a result, many existing works about LLMs’ performance in games (e.g., [1], [2]) only focus on a specific game and cannot be extended to other games or scenarios. Indeed, one contribution of DynaEval is that it provides a standardized modeling method of all kinds of games using “interaction goals” and “interaction rules” and provides a general framework for running all these games and evaluating the ability of LLMs, as mentioned in A1.b. With this framework, researchers need not waste their time on designing a specific system for running a specific game, but focus more on the properties of the game itself such as what to evaluate (which can be reflected in the interaction goals and interaction rules).

W3.b. The games seem odd, and it's not obvious what capability of the model they are trying to measure. For example, I'm not even sure it makes sense for the Code G&R and the Machine Translation games to actually be turn-based games. As I said, I do see why games can be useful, but the devil is in the details.

A3.b. Thank you for your doubt. As mentioned above, DynaEval is a general framework for running all kinds of games and evaluating LLMs, and “what ability to measure” depends on the design of games, which is decided by users of DynaEval. Generally, we can categorize the abilities of LLMs into “fundamental” abilities and “domain-specific abilities”. Fundamental abilities are necessary for all kinds of dynamic interactions between LLMs, such as the ability to understand and reasoning. Domain-specific abilities are required by specific domains or tasks, such as the ability to generate logically correct and well-styled codes in code generation. The design of evaluation tasks can decide the domain-specific abilities to evaluate. For example, as introduced in Section 2.4, p6 in our paper, the Code G&R evaluates the code generation ability of programmer LLMs and the code review ability of reviewer LLMs. In real-world scenarios, users of DynaEval can design their own evaluation tasks to evaluate their required domain-specific abilities of LLMs.

[1] Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, & Yang Liu. (2023). Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf.

[2] Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, & Gao Huang. (2023). Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation.

We would like to thank you very much for the detailed feedback and valuable suggestions.

W1. The proposed DynaEval framework lacks a specific evaluation goal since the "ability of interaction" is too big.

A1. Thank you for the question. We agree with the opinion that LLM evaluation should focus on specific abilities. As for DynaEval, what should be noticed is that the ability to evaluate depends on the design of the evaluation task, which is determined by the user of DynaEval. On the other hand, DynaEval itself provides a platform/framework for users to design and implement their evaluation task that evaluates the task-specific abilities of LLMs in dynamic interaction scenarios. For example, in our experiment, the Code G&R evaluates the code generation ability of the programmer LLM and the code review ability of the reviewer LLM. The reviewer may doubt that this task cannot disentangle the understanding and reasoning ability of LLMs because they are also required by this task. However in real-world dynamic interaction scenarios, LLMs always need to use multiple abilities to solve users’ problems, and the understanding and reasoning ability is the basis of them because they need to optimize their solution based on their understanding and reasoning of user feedback. Therefore, as Code G&R and Machine Translation, users of DynaEval can design their own evaluation tasks to evaluate task-specific or domain-specific abilities of LLMs. In our paper, we also introduced in Appendix A.4 how to design evaluation tasks based on users’ own requirements.

W2. The paper tries to standardize the interaction process and spends many words to emphasize the fairness and stability, and hence propose a synchronous interaction algorithm which ensures anonymity and multiple independent runnings. However, I think most of them are common senses in many evaluation tasks.

A2. Thank you for your question. Indeed, fairness and stability are only easy to achieve in traditional supervised signal-based evaluation or human-based evaluation because the response of each LLM is independent of and isolated from others. However, In the dynamic interaction between LLMs, things are much more complicated. LLMs might expose their real identity because of a lack of supervision, and the synchronicity is easy to be violated because the response time of different LLMs can vary a lot. Moreover, the synchronicity also depends on game rules. For example, in the public goods game, LLMs are required to respond simultaneously in each round. But in Code G&R, LLMs are required to respond one by one. In real-world scenarios, the game rule can be much more complex. As a result, to standardize the complex interaction process, we propose the referee to help guarantee the fragile but significant anonymity and rate LLMs’ performance, and we propose the message pool which uses the synchronous interaction algorithm to compulsively guarantee the synchronicity of the dynamic interaction process.

W3. Third, the paper tries to build a close relation between DynaEval and extensive-form games with perfect information (EGPI). But I did not see necessity of doing this.

A3. Thank you for your question. As mentioned in A2, the dynamic interaction process between LLMs is much more complicated than the traditional supervised signal-based or human-based evaluation process. As a result, we utilize the EGPI to standardize the interaction process in all kinds of tasks and scenarios and introduce the fairness and stability condition of DynaEval. As for shared properties of DynaEval and EGPI such as the existence of equilibrium (Do you mean Nash equilibrium?), since we have proved that DynaEval belongs to EGPI, it is theoretically easy and convenient to analyze the property of DynaEval in terms of game theory. As this paper mainly focuses on LLM evaluation in dynamic scenarios, such an analysis exceeds the range of research of this paper. We plan to analyze the theoretical properties of DynaEval in our next research.

Q1. In the example of IDIOMS SOLITAIRE, which specific ability are you testing? From my point of view, winning LLM should have seen more idioms so that it can come up with an answer easily. But does this has anything to do with "interaction"?

AQ1. As introduced in Section 2.4, p6 in our paper, we test the ability to retrieve and use the Chinese vocabulary of LLMs. Interaction is significant in this task, because LLMs’ responses are based on the context, and Chinese idioms are limited and it is easy for LLMs to acquire a host of idioms. If two participants have seen a similar amount of idioms, they need to use specific strategies to win the game, such as coming up with a Chinese idiom with the last character complex and rare, which makes it hard for the opponent LLM to come up with a valid Chinese idiom, even though it has seen a lot of Chinese idioms.

We would like to thank you very much for the detailed feedback and valuable suggestions.

W1. There needs to be an explicit and clear comparison between the dynamic evaluation results and what static evaluation methods find with an accompanying explanation.

A1. Thank you for your constructive suggestion. Indeed, we conducted the machine translation task on different language pairs. Due to the page limitation, we only put the EN-ZH in the body of the paper and put the rest in Appendix A.9, p20. According to our experiment, ChatGPT only outperforms GPT-4 in the EN-ZH setting of machine translation, while underperforming GPT-4 in the DE-EN and EN-FR setting. Similarly, in the Idioms Solitaire task (which is also relevant to Chinese), we also observed that ChatGPT outperforms GPT-4, as shown in Table 1 and Table 2. These results might demonstrate that ChatGPT can perform better than GPT-4 in Chinese-related tasks. We will discuss more about this discovery in the next version of our paper.

W2. They discuss the “improvement of performance” through the evaluation, even while highlighting their main results in the conclusion. While LLMs can improve performance on tasks through interaction, this observation seems completely beside the point of DynaEval which is an evaluation procedure, so the absolute performance of the LLMs is not important.

A2. We suppose that the “ability to improve performance through interaction” is a significant part of the performance of LLMs. As mentioned in the third paragraph of Section 1, in dynamic interaction scenarios, LLMs repeatedly get feedback and optimize their output to gradually meet users’ requirements. As a result, “how is the performance of an LLM after getting feedback” is significant in our evaluation. In addition, since the feedback highly affects the degree of improvement and is provided by another agent, “how an LLM can improve the performance of another LLM” is also significant in our evaluation.

Q1. Is the stability condition reasonable for LLMs? Couldn’t they not converge? Aren’t those sorts of guarantees made for substantially similar agents?

AQ1. The stability condition is reasonable and indispensable for LLMs. Due to the uncertainty in the text generation of LLM and the variety and complexity of interaction environments (e.g., different interaction contexts can lead to different input for an LLM), the uncertainty of interaction results widely exist in dynamic interaction scenarios and can highly affect the evaluation of LLMs. Therefore, we noticed that the performance of an LLM should be modeled as a random variable rather than a specific value. As a result, we utilize the stability condition of DynaEval to evaluate the expected performance of LLMs, as shown in Eq.(1) in our paper. With multiple running of an evaluation task, the performance of participant LLMs can converge in distribution. This condition is also reasonable for dissimilar agents. As shown in Eq.(2), when modeling the full distribution of an LLM’s performance, the interaction histories (and other LLMs’ abilities) are only viewed as the context and condition of the random variable of the LLM’s performance and can be eliminated using the law of total probability.

Q2. The source code link does not work.

AQ2: Sorry for the confusion. Our source code is available at https://anonymous.4open.science/r/DynaEval-D334 . We will update the source code link to the new version of our paper.

Thank you for the responses.

However, I have decreased my score with the rebuttal because there is definitely confusion about what this relative performance evaluation is compared to static evaluation. When you clarified this in the first rebuttal comment, you still are replied to total score between 2 LLMs only in the dynamic setting. I still am in support of this paper, but for it to be a meaningful benchmark, it needs to clearly find that comparing 2 LLMs (i.e. ChatGPT and GPT-4) yields different results in this test than the existing ones already defined in prior work.

It is critical that you very definitively point out "on [this task (for example machine translation)], static evaluation methods yield [this relative performance] between 2 LLMs and our dynamic evaluation method uncovers [this relative performance]". Please clearly write this for all tasks for which the relative performance is not the same as a comment (ultimately to be included in the paper) for me to fully support it in further discussion.

W3. For the fairness evaluation of LLMs, it is unclear whether it is possible to develop referees automatically in a supervised signal-based manner with less human-based guidance.

A3. First, it is hard to develop referees automatically in a supervised signal-based manner in DynaEval because the interaction environment is complex, and the interaction rule and evaluation metric of different dynamic interaction-based evaluation tasks can vary a lot in real-world scenarios. As a result, it is inefficient to customize a supervised signal-based referee for each (countless) evaluation task. Moreover, during the interaction process, participant LLMs may generate abnormal outputs (such as outputs that violate the format) which affect the running of the evaluation process and should be distinguished by the referee. However, supervised signal-based referees cannot distinguish these abnormal outputs. Therefore, supervised signal-based referees are hard to develop in DynaEval.

Second, human-based guidance for referees does not affect the fairness of LLM evaluation. As illustrated in Condition 1 in our paper, all participant LLMs should be anonymous during the interaction process. This condition means that participants are not only anonymous to each other, but also anonymous to the referee. That is, the referee can only identify each participant by their assigned ID. As a result, the referee can only evaluate the performance of LLMs based on their interaction history, which is identity-irrelevant. As a result, the fairness of LLM evaluation can still be achieved when using referees with human-based guidance.

W4. The discussion on the proposed evaluation method compared to existing ones is quite weak. It is unclear how the proposed one can tackle the dynamic nature of the LLMs interaction while the existing evaluation methods cannot.

A4. As presented in the third paragraph of Section 1, we discussed the reason why existing evaluation methods cannot tackle the dynamic nature of the LLMs interaction. In the fourth paragraph, we further illustrated how the proposed DynaEval can solve this problem. For traditional supervised signal-based methods, their evaluation environments are static, and hard to capture the complexity of dynamic real-world situations. Moreover, although some evaluation methods support multiple-round conversations, they still fall short in providing a dynamic interaction environment and lack generality.

We would like to thank you very much for the detailed feedback and valuable suggestions.

W1. The technical contribution of this paper is limited because the method fails to come up with a practical and novel fairness condition or stability condition. Also, the concepts of referee and message pool are not novel either.

A1. Thank you for your doubt. The technical contribution of this paper is that: 1. We noticed and proved the relationship between the dynamic interaction process of LLMs in real-world scenarios and EGPI; 2. Based on EGPI, we proposed DynaEval which is a general framework for dynamic LLM evaluation that can cover most interaction processes in real-world scenarios, which is much more complex than traditional signal-based evaluation methods. Anonymity, synchronicity, and independent runs are fundamental yet indispensable in all dynamic interaction-based LLM evaluation. However, they are only easy to achieve in traditional supervised signal-based evaluation or human-based evaluation because the response of each LLM is independent of and isolated from others. In the dynamic interaction between LLMs, things are much more complicated. LLMs might expose their real identity because of a lack of supervision, and the synchronicity is easy to be violated because the response time of different LLMs can vary a lot. Moreover, the synchronicity also depends on game rules. For example, in the public goods game, LLMs are required to respond simultaneously in each round. But in Code G&R, LLMs are required to respond one by one. In real-world scenarios, the game rule can be much more complex. As a result, we propose the referee to help guarantee the fragile but significant anonymity and rate LLMs’ performance, and we propose the message pool which uses the synchronous interaction algorithm to compulsively guarantee the synchronicity of the dynamic interaction process.

W2. The task rule itself has a lot to do with fairness and stability for the evaluation task, but very little details about it has been revealed.

A2. Thank you for your question. As illustrated in Section 2.2 and Definition 1, the task rule of a dynamic interaction-based evaluation task consists of an interaction goal and an interaction rule. Such a design is simple yet sufficient for depicting all dynamic interaction-based tasks in real-world scenarios.

As for fairness and stability, they are disentangled from task rules, and achieved by the special mechanism of our proposed DynaEval (referee, message pool, etc.) that is irrelevant to specific task. This design is in alignment with real-world requirements. For example, in the review-rebuttal process of this paper, the review-rebuttal rule only regularizes the interaction goal and the interaction rule for reviewers and authors. For reviewers, the goal is to accept high-quality papers and reject low-quality papers while helping improve the quality of the latter. For authors, the goal is to make their paper accepted by the conference and improve its quality. The interaction rule is that reviewers first review the paper and give the comment. Then authors and reviewers can discuss the paper during the rebuttal phase. During this process, stability is ensured by assigning multiple reviewers to a paper, and fairness is ensured by the surveillance of the OpenReview platform. Indeed, even though the task rule does not regularize fairness and stability, they can still be satisfied by the special mechanism of DynaEval.

Thanks for your approval to our work. We will discuss the work on multi-LLM agents for the study of cooperative interactions that you mentioned above and follow more similar prior works in the future.