TIGERScore: Building Explainable Metric for All Text Generation Task

There could be more details provided on the prompt engineering strategies used to create MetricInstruct.

Thank you for your suggestion to discuss prompt engineering strategies in greater detail. We agree that sharing this information can assist other researchers in crafting more effective prompt templates for their specific use cases. Consequently, we have thoroughly discussed various strategies for eliciting knowledge from GPT-4. Here, we provide a summary of these prompting strategies:

When creating evaluation aspects of each task (details in appendix A.1) :

• We use GPT-4 to generate candidate aspects, and then select a few of them as the final aspects with human efforts.
• We prompt GPT-4 to generate concise definitions and typical specific error types under each aspect, and then revise them with human efforts.

When designing prompting templates to get the error analysis from GPT-4 for each task, we adopt strategies (details in appendix A.2):

• Two-Step Generation and Formatting Process
• Incorporation of Task-Specific Words in Initial Queries
• Integration of Predefined Aspect Definitions for Focused Evaluation
• Classification of Errors Using Major/Minor Error Guidelines
• Adopting a 0.5 to 5 Scale for Scoring Error Severity

Dear Reviewer A6PK:

Thank you for your insightful review. We are glad to hear that you appreciate TIGERScore as a transparent and trustworthy metric and achieve state-of-art correlation metrics with human judgments

According to the feedback of all the reviewers, we have added the corresponding experiments in the revised paper with supplementary experiments in the appendix. We have summarized the major updates in General Response (shown in a separate comment on this page).

We would like to address your concerns in detail below.

The generalizability to open-ended generation like storytelling seems limited currently. More investigation into enhancing this is needed.

Thanks for pointing out the TIGERScore V1’s problem of generalization ability. We also noticed these weaknesses after the V1 development and have been investigating the reason behind them.

Our investigation shows that the low scores on story generation are caused by the lack of diversity in the dataset. Though the MetricInstruct V1 has included various text generation tasks and broad error types in training instances, they are still limited to a specific task.

To increase the generation ability of TIGERScore, we collect 10k more instruction-following data from alpaca-52k and apply the same data preparation pipeline querying GPT-4 to get high-quality error analysis responses. We train TIGERScore-V1.2 on them and the results show that the generalization ability has been increased greatly (see results in Appendix A.7 )

There is no comparison of computational efficiency with other metrics. Efficiency can be a practical concern.

Thank you for highlighting the importance of computational efficiency. We admit that evaluation efficiency is vital, particularly when evaluating large-scale datasets where speed determines user preference.

TIGERScore, being a 7B Llama-based Large Language Model (LLM), does face challenges with inference speed. However, with the aid of the VLLM toolkit on a single A800 GPU, TIGERScore efficiently processes a single instance in approximately 0.2 seconds.

In comparison, traditional metrics, such as BLUE, BERTScore, etc. are quicker, ranging from 0.1 to 0.001 seconds per instance. Nonetheless, other explainable metrics with large model sizes, like InstructScore and PandaLM. also encounter speed constraints. Traditional metrics are faster because their scores are unexplainable. TIGERScore, as an explainable metric, provides not only the score with higher correlations but also the structured error analysis content.

Therefore, we believe that TIGERScore's processing time of 0.2 seconds per instance aligns well with the current industry requirements, especially in scenarios where interpretability is of great significance.

The correlation gains over prior metrics, while substantial, are incremental in some cases. The gains do not feel dramatically superior.

Thanks for pointing out your concern about the incremental gains. We understand your concern and summarize the following points in response:

• As a reference-free metric, it’s naturally hard to achieve similar correlation performance with reference-based metrics, especially in cases where references are of high quality. We contend that one of the values of TIGERScore is its substantial gains compared to other reference-based metrics.
• As shown in our TIGERScore V2 results, the correlation has surpassed all other reference-free metrics and achieved approaching performance with the GPT-4-0-shot. One exception is COMETKiwi, which has been specifically trained for translation on human eval rating. Therefore, it’s acceptable that TIGERScore as a more general evaluation metric fails to surpass it.
• Except for correlation, one significant feature of TIGERScore is the structured error analyses and explanations it outputs, which most other metrics cannot provide.

Therefore, we conclude that our current TIGERScore correlation has made great improvement on correlations compared to existing reference-free metrics and approached the performance of GPT-4-0-shot. And contrast to most other metrics, TIGERScore is explainable.

We hope our response has solved our concern!

Q5: Table 6 - do authors have any intuition why there is decreased performance for Data2Text when all tasks are included?

Thank you for your insightful question!

Firstly, for the decreased performance for data2text when all tasks are included in TIGERScore V1, we tend to explain it that:

• Data2text differs from other tasks: while other tasks utilize natural language, the source inputs of data2text are mostly structured knowledge tuples. Therefore, it might benefit little from the other knowledge through multi-task learning
• Multi-task learning usually exhibits some extent of performance tradeoff across different tasks. And we think data2text happens to be the victim.

Secondly, we want to clarify that we have conducted a brand-new ablation study based on our TIGERScore V1.2 dataset (see Table 6 or the following table), and results show all tasks except mathQA benefit a lot from multi-task learning. We blame the decrease of mathQA after multi-task learning for its inherent difficulties.

Q6: Fig 3 and Table 6 - is there any specific reason why the results on MathQA and StoryGen are excluded? It would be nice to place the Figure and Table together for better comprehension.

For mathQA, we didn't finish training the model for the math QA task due to time and computing power constraints, thus excluding it from the ablation results.

For the story generation task, we did not do an ablations study on it because of our negligence. We only focus on conducting experiments on story generation for the main correlation results. It's a great suggestion to include them in the ablation study for better comparison.

Therefore, in the current version, in response to your suggestion, we have included all the results of mathQA and story generation in the ablation studies. Results show that the combination of real-world data and synthetic data, along with mixed training on all tasks, has been found to enhance the performance of TIGERScore in most cases compared to individual training. You can refer to more details in Section 4.5 of our revised paper version.

We hope our response answers your concern well!

It is not clear why the model's max context length is set to 1024 when it can ingest 4096 tokens.

Thank you for your insightful question!

Our decision to set the context length at 1024 is primarily driven by hardware limitations. Currently, we lack the GPU resources necessary to fine-tune a Llama Model with full-context length, as this task demands more than 8 A100 GPUs.

Additionally, our analysis of the dataset, detailed in Appendix A.5, reveals a significant finding: over 90% of the training instances have sequence lengths below 1024, with nearly all falling under 1250. This data strongly supports our choice of a 1024 context length for both fine-tuning and testing purposes.

We hope this explanation clarifies your query and proves helpful.

Thank you for your suggestion. Due to budget constraints, each item undergoes only one round of human evaluation, and as a result, we do not record inter-annotator agreements.

Our evaluator is selected from Prolific, a well-known crowdsourcing platform. To become an official annotator on Prolific, individuals must successfully pass specific tests. We periodically sample and review these annotations. Any poor-quality annotations, along with all other work by the responsible annotators, are excluded. This approach helps us maintain the quality of our evaluations.

In summary, we are confident in the validity of our human evaluation results. This confidence stems from the high quality of our annotators and the rigor of our review process. We hope this explanation addresses your concerns.

There are no results related to hallucination or factual correctness of the generated text, which would have made the results more compelling

Thanks for your construction suggestion. In response to your concern, we have conducted two experiments to investigate the hallucinations of the generated text.

Firstly, as suggested by reviewer WhZB, we run TIGERScore to evaluate the gold reference for each task to see whether TIGERScore will hallucinate errors on non-error outputs. Results are shown in the following table. Here for each task, the $0$ column refers to the percentage that TIGERScore reduces 0 scores for the gold references. The $> -2$ column refers to the percentage that TIGERScore reduces less to 2 score on the gold references.

On average, TIGERScore (13b) does not produce any errors in the gold reference sentence about 77.21% of the time. In these instances, it either makes no errors or only minor ones, resulting in a score reduction of less than 2.0 (minor error). Therefore, while TIGERScore may occasionally hallucinate errors, its overall rate of hallucination is deemed acceptable.

Secondly, for each task, we ran TIGERScore on 20 samples with errors in the system output. We then used GPT-4 to determine if these samples contained hallucinations or factual inaccuracies (see Appendix A.7).

According to the results above, approximately 89.28% of TIGERScore's error analyses were free from hallucinations or factual errors. We acknowledge the limitations of our study, including the small sample size and the reliance on GPT-4 rather than human evaluators. Nonetheless, our findings are significant, demonstrating that TIGERScore is effective at avoiding hallucinations in generated content.

We hope our response answers your concern.

The paper does not extensively explore scenarios where TIGERScore might fail to provide accurate error analyses. Understanding its limitations could be valuable for users. The paper briefly mentions the hardware used for fine-tuning but lacks an in-depth discussion of the computational resources required for implementing TIGERScore.

Thank you for your constructive suggestion.

We acknowledge the importance of addressing the study's limitations and have added a new section in Appendix A.6 to discuss the limitations of TIGERScore. This section covers issues in the following:

1. There are occasional generation of hallucinated errors by TIGERScore, and a need for further verification of the quality of its generated explanations.
2. We recognize that despite TIGERScore's ability to evaluate an instance in 0.2 seconds using VLLM, there remains a performance gap when compared to traditional metrics like BLEU and BERTScore.
3. We explore the discrepancies between local errors and global evaluation, concluding that more refined methods of accumulation are needed.

Regarding the computational resources required for TIGERScore, we provide a detailed discussion in Section 2.4. This includes the use of 8 A100 GPUs for training TIGERScore-13b and 4 A100 GPUs for TIGERScore-7b. We also note that TIGERScore can be operated on a single A100 or A6000 GPU, again with the aid of VLLM, achieving an inference speed of approximately 0.2 seconds per instance. We hope that these additions adequately address your concerns.

Is there any curriculum used for training on the different tasks?

No, we simply mix the data from all the tasks and perform the default random sampling during the training process.

One idea that might be interesting, but we did not try due to the limited resources, is first finetuning on the first 5 well-defined tasks, i.e. summarization and translation, etc, and then do the instruction-finetuning on the instruction-following dataset. We leave this for further research.

Thank you for your insightful review. We are delighted to hear that you recognize TIGERScore's significance on NLG evaluation and the value of MetricInstruct.

According to the feedback of all the reviewers, we have added the corresponding experiments in the revised paper with supplementary experiments in the appendix. We have summarized the major updates in General Response (shown in a separate comment on this page).

We would like to address your concerns in detail below.

Even though the results are promising at the initial level, the spearman correlation of TIGERScore is lower than the other existing metrics for 4 out of the 7 tasks (Fig 1). Table 4 also shows a huge difference between the reference-free and reference-based metrics which hinders its usage as a universal metric (main contribution of the paper).

Thank you for pointing out the potential weaknesses and we totally understand your concern.

We first talk about the correlation performance. Our newly delivered TIGERScore-13b-V1.2 reaches the best compared to all the existing reference-free metrics on 5 out of 7 tasks, except that for summarization and translation. However, TIGERScore maintains small gaps to the highest correlation scores. For example, in the translation task, TIGERScore-13b-V1.2 reaches 44.99, only 3.99 to the best one in translation.

Besides, we contend that Spearman should not be the only correlation metric to focus on, as suggested by reviewer 2ZJr. On Kendall metric (see Appendix A.7 Table 20), TIGERScore-13b-V1.2 reaches the best across all the tasks compared to all other reference-free metrics, even surpassing GPT-4-0-shot on 4 out of 7 tasks. Therefore, we contend that the current correlation performance of TIGERScore is super competitive.

For your second concern on the gap between reference-based metrics, we have raised 3 important key features in section 1 that universal evaluation metrics should try to tackle, which are 1) Dependency on references, 2) Limited to specific domains, and 3) Lack of attribution. Compared to existing reference-based metrics:

1. The reference feature of TIGERScore makes it easier to quickly apply to multiple scenarios, without first finding a reference.
2. The reference-based feature achieves low overall correlation performance across multiple tasks. Considering Spearman, the best traditional reference-based metrics get 34.16 in average performance, while TIGERScore reaches 38.60 (7b) and 41.52 (13b) respectively, which is of superior advantage.
3. TIGERScore is explainable through the generated error analysis, while few of the reference-based metrics are explainable (only InstructScore)

Therefore, considering all the above factors as universal metrics, we believe TIGERScore is a more qualified one than the reference-based metrics.

Suggestions/Comments: Fig 1 caption: shows te -> shows the Section 5.2: Research on this fields -> in these fields Please take care of the quotes in Section 2, Table 3 caption Please specify and distinguish between LLaMA and LLaMA2 in the abstract and the main paper.

Thank you for your writing suggestions. We have corrected the typo in Figure 1, the grammatical error in Section 5.2, and the issue with quotes in Section 2. Additionally, we have updated all mentions of 'Llama' in the paper to 'Llama-2', which is the specific version TIGERScore is fine-tuned on.

Would the corresponding code be made available which is necessary for reproducibility.

Thank you for your interest in the TIGERScore code repository. We are committed to the principles of open-source community conduct and have made all the code available on GitHub.

You can find further details about our project on the website listed in our paper. Our efforts have been focused on enhancing the user-friendliness and accessibility of our code repository. Additionally, we are in the process of integrating TIGERScore into the Hugging Face evaluation metrics to better serve the research community.

Weakness 3: It would be nice to have a comparison with 0-shot GPT-4 based evaluation (having limited computational complexity) which has also been recently used as a reference free and explainable metric [1,2].

Question 1: Have the authors also experimented with InstructScore which seems to be very related to the proposed approach?

Question 3: Did the authors experiment with 0-shot abilities of the LLaMA-2 (both chat and non-chat) models as a reference-free evaluation using prompts?

Thank you for your valuable suggestion. We agree that incorporating more robust baseline comparisons provides a clearer understanding of TIGERScore's strengths and limitations. Consequently, we have included 0-shot GPT-4 and 0-shot LLaMA2-Chat as potent reference-free baselines in Table 4. During our evaluation of the LLaMA-2-non-chat variant, we observed its inability to follow instructions to output valid scores, thus leading to its exclusion from our baseline analysis. Additionally, InstructScore is added as a powerful baseline a reference-based baseline.

Our findings reveal that TIGERScore outperforms LLaMA-2-Chat across all tasks and delivers comparable results to GPT-4, demonstrating its impressive capabilities. Furthermore, TIGERScore exceeds InstructScore in all tasks except for translation, where the performance gap of translation is minimal. This slight disparity in translation is justifiable, considering InstructScore is a reference-based metric, whereas TIGERScore operates as a reference-free metric.

A worth-noting gap between GPT-4-0-shot performance is on mathQA. We believe this severe gap is caused by 1) lack of math-specific fine-tuning, 2) inherent hardness of the mathQA task. Obviously, further work is to be explored on mathQA evaluation.

Thanks for your valuable suggestions on the writing!

• In response to your suggestion 1, we have referred to Figure 2 and Table 5 in the corresponding text sections that discuss their results.

• We have addressed suggestions 2, 6, 7, and 9 by correcting the typographical errors.

• Following your suggestion 3, we have moved the original Section 4.2 to Section 2.4, adding more details about TIGERScore's implementation in Section 2 as you requested.

• As per your suggestion 4, we have included references to the tables immediately following this sentence.

• For your suggestion 5, We have revised this sentence and excluded the existence of LD distance to make it more clear. The revised one is:

In advance, to mitigate the impact of the hallucination in error analysis, we split the error location and the source by spaces. Then we remove error positions that contain too many words that do not appear in the input source to avoid the illusion of error locations.”

• In line with suggestion 8, we have incorporated the results of TIGERScore V1.2. This update demonstrates a significant enhancement in StoryGen's performance, underscoring our improved generalization capabilities. Therefore, we believe the statements still hold well.

• For suggestion 10, we have added simple phrases after each number indicating how we calculated them.

Thank you for your inquiries regarding our prompting template design. Given that questions Q2 through Q7 all relate to this topic, we have opted to collectively address them in Appendices A.1 and A.2, where we discuss the various prompting strategies employed in the template creation process. Below, we provide a summarized response to each question.

Question 2: How are the prompt templates in Appendix 2 created? There are minor variations from task to task, is this intentional (e.g., the way minor and major mistakes are explained in the prompts changed from template to template)?

The templates discussed in this paper were developed from our practical experience and intuition, as detailed in Appendix A.2. They all stem from a single base template, with minor variations introduced to incorporate task-specific terminology, aiming to influence GPT-4's responses in a certain way.

Furthermore, to assess the effectiveness of these templates, we conducted preliminary experiments focusing on their correlation. During the development process, each template underwent several iterations by a designated author, leading to the final versions presented in this paper.

Question 3: Why explain major and minor mistakes in the prompts, when you can ask the model to infer from the aspect list? Is this for providing two criteria (major/minor label, and penalty score) to filter unreasonable candidate errors?

This explanation is inspired by the MQM translation human rating system and InstructScore prompting template. Our intuition is that this classification helped GPT-4 in assigning more consistent scores to each error, countering its instability with numerical judgments.

Question 4: Some of the templates (e.g., Table 11) are missing prefixes like "References:", and "Task instructions:", is this intentional or a typo?

No, As stated in our response to question 2 and Appendix A.2, these templates are the results of a few iterations by our authors, together with some intuitions. For example, in tasks like mathQA and instruction-following, where the context is self-explanatory, we omitted specific keywords like “Task Instruction” and “reference”.

Question 5:. Why is $aspect_list not used in translation and data2text prompt templates?

As stated in Appendix A.2, the incorporation of ${aspect_list} is used to address the issue where GPT-4 might miss crucial evaluation aspects and overlook some errors that we think are important. Exceptionally, in data2text tasks, we found that asking GPT-4 to directly evaluate errors was sufficient, and thus, we did not include our predefined aspects.

Question 6:. Are the definitions of evaluation aspects ($aspect_list) for each task created based on consensus or by the authors?

As stated in Appendix A.1, these aspects are created with the assistance of GPT-4 and human revision. They are designed to be mutually exclusive and collectively exhaustive for a single task. The steps to create these aspects are:

• Step 1: We prompt GPT-4 to output 20 candidate aspects for each task.
• Step 2: We ask GPT-4 to summarize these aspects into 3 to 5 general aspects for this task.
• Step 3: We ask GPT-4 to generate the detailed definition and 5 specific error types under each aspect.

Then after final human revision by our authors, these aspects are created.

Question 7:. Why is a scale from 5 to 0.5 chosen? Any special reason?

In Appendix A.2, we initially employed an integer scale ranging from 1 to 5 for error penalty scores. This scale proved effective for translation tasks. However, for other tasks such as summarization, data-to-text generation, and instruction-following, it was less effective. Our experiments showed that a more nuanced scoring scale, ranging from 0.5 to 5, resulted in a better correlation across all tasks.

Question 1: How many system outputs are correct (no errors)? How well does TIGERScore perform on those outputs? Would it hallucinate errors frequently? Have you tried running TIGERScore on reference outputs?

Thank you for your insightful question. We'll address your queries sequentially.

To begin with, referencing our earlier response to weakness 1, you'll find a detailed analysis in Appendix A.5. The data indicates that approximately 20% of the system outputs in the training dataset are entirely error-free.

Furthermore, to assess the effectiveness of TIGERScore on correct outputs and its potential for generating hallucinated errors, we have constructed a collection of reference outputs to create new test datasets for each task, as you recommended. Results are summarized below (or see Appendix A.7 Table 21).

Here for each task, the $0$ column refers to the percentage that TIGERScore reduces 0 scores for the gold references. The $>-2$ column refers to the percentage that TIGERScore reduces less to 2 score on the gold references.

On average, TIGERScore (13b) does not produce any errors in the gold reference sentence about 77.21% of the time. In these instances, it either makes no errors or only minor ones, resulting in a score reduction of less than 2.0 (minor error). Therefore, while TIGERScore may occasionally hallucinate errors, its overall rate of hallucination is deemed acceptable.

Besides, for each task, we ran TIGERScore on 20 samples with errors in the system output. We then used GPT-4 to determine if these samples contained hallucinations or factual inaccuracies (see Appendix A.7).

According to the results above, approximately 89.28% of TIGERScore's error analyses were free from hallucinations or factual errors. We acknowledge the limitations of our study, including the small sample size and the reliance on GPT-4 rather than human evaluators. Nonetheless, our findings are significant, demonstrating that TIGERScore is effective at avoiding hallucinations in generated content.

We hope our response answers your concern well!

Weakness 1: lack of discussion on the distribution of quality in the provided dataset. For example, it is unclear how TIGERScore would perform on notes with no errors or a ranking scenario.

Thank you for your valuable suggestion regarding the analysis of our dataset distribution. In response, we have included a comprehensive analysis in Appendix A.5. This addition aims to enhance our understanding of MetricInstruct for our users.

Specifically, we have analyzed the context lengths of training instances to determine the most appropriate model context length. Furthermore, we have investigated the frequency of errors in each instance across all tasks. Our findings indicate a natural distribution of errors. It’s crucial that a number of instances with no errors exist in the dataset because it helps the model learn what is correct and reduces fabricated errors from the model.

Dear Reviewer WhZB:

Thank you for your insightful review. We are delighted to hear that you recognize TIGERScore as a competitive reference-less evaluation metric of high significance.

According to the feedback of all the reviewers, we have added the corresponding experiments in the revised paper with supplementary experiments in the appendix. We have summarized the major updates in General Response (shown in a separate comment on this page).

We would like to address your concerns in detail below.

Q1: Have other correlation metrics, like Pearson and Kendall rank correlation coefficients, been considered? These metrics might reveal different aspects of the correlation between model outputs and human preferences.

Thank you for your valuable suggestions. We did record the results of Pearson and Kendall correlation results in all of our experiments. To provide a comprehensive view, we have included the correlation results of all three metrics (Pearson, Kendall, and Spearman) in Appendix A.7. Our findings show that the results from Pearson and Kendall analyses are consistent with those from Spearman reported before, affirming that TIGERScore correlates effectively with human ratings across various correlation aspects.

Weakness 2: Regarding human evaluation, there's ambiguity about the identity of the 'human experts' and the reliability of their responses. The overall rating of TIGERScore is noted as 70.8%, but it's unclear whether this score is deemed acceptable or not. It would be beneficial to compare TIGERScore with other existing explainable metrics in Section 5.2, demonstrating the differences from the baselines.

Question 2: Could you clarify who the 'human experts' are for the human evaluation?

Thank you for your constructive suggestion!

To clarify, "human experts" refers to evaluators chosen from Prolific, a renowned crowdsourcing platform. These individuals must pass certain tests to be approved as official annotators on Prolific. We maintain high evaluation standards by periodically sampling and reviewing these annotations. If any annotation is found to be of poor quality, it is removed from our system along with all other work by the responsible annotator. This practice is key to upholding the integrity of our evaluations.

Regarding the TIGERScore, as mentioned in Section 1, it stands out as one of the few metrics that is both explainable and applicable across various tasks. Unlike metrics like UniEval and GPTScore, which are considered explainable due to their multi-aspect scoring capabilities but lack text explanation capabilities, TIGERScore offers contextual text explanations. While PandaLM can generate text explanations, they are limited to simplistic overall assessment sentences, providing minimal information. Moreover, InstructScore is confined to translation tasks, making it non-comparable across different tasks. Due to these distinct features, we did not conduct a direct comparison of explanation quality with baseline metrics.

Lastly, the 70.8% overall positive rating pertains specifically to the feedback that "TIGERScore’s explanations provide helpful revision suggestions." This suggests that TIGERScore's explanations are perceived as beneficial around 70.8% of the time. While determining if a 70.08% rating is acceptable can be subjective and lacks a comparative context, we believe this percentage does affirm the absolute helpfulness of TIGERScore.

TIGERScore exhibits limitations, particularly evident in held-out tasks (story generation), raising doubts about the claim of being a universal metric for all text generation tasks in the title. The title prompts questions about its efficacy in especially evaluating open-domain conversation tasks, given the considerable diversity in model outputs.

Thanks for pointing out the concern of TIGERScore’s limitations in held-out tasks and we totally understand your concern. We noticed these weaknesses after the V1 development and have always been investigating the reason behind them.

It’s shown that the TIGERScore-V1’s limitations on generalization ability are caused by the lack of diversity of tasks in the datasets. Though the MetricInstruct V1 has included various text generation tasks and broad error types in training instances, they are still limited to a specific one, except for the instruction-following task.

In TIGERScore V1.2, we value the importance of instruction-following tasks, which we believe is important in increasing the generation ability of TIGERScore. Therefore, we collect 10k more instruction-following data from alpaca-52k and apply the same data preparation pipeline querying GPT-4 to get high-quality error analysis responses. We train TIGERScore-V1.2 on them and the results show that the generalization ability has been increased greatly.

We now have trained a new version of TIGERScore-V1.2, which gets great performance enhanced in generalization ability, shown by the increased correlation score in the story generation task. TIGERScore V1.2’s results have been updated in Table 4. A summarized difference between V1.

From the table, we can see clear improvements in data2text, instruction-following, and the held-out story-gen task. And still maintain similar performance with V1 on summarization, translation, and long-form QA tasks.

Dear Reviewer 2ZJr :

Thank you for your insightful review. We are glad to hear that you appreciate TIGERScore as a robust and adaptable NLG evaluation methodology.

According to the feedback of all the reviewers, we have added the corresponding experiments in the revised paper with supplementary experiments in the appendix. We have summarized the major updates in General Response (shown in a separate comment on this page).

We would like to address your concerns in detail below.