ER2Score: An Explainable and Customizable Metric for Assessing Radiology Reports with LLM-based Rewards

Thanks for your feedback, to address your concerns, we have outlined the following points:

We have to point out our approach represents a significant improvement in providing human-consistent evaluation metric for R2Gen task: Our Kendall Tau's correlation improves from 0.63 (the second best) to 0.75, and our Spearman correlation improves from 0.81 (the second best) to 0.91 on ReXVal dataset. Please note that, to advance R2Gen research, it is essential to develop evaluation metrics that are highly correlated with human assessments. Currently, many methods still focus on improving existing non-trainable metrics, but these metrics often have weak correlation with human evaluations. Relying on them as a guide for improvement may misdirect R2Gen research. Therefore, constructing metrics closely aligned with human judgment not only enables more accurate quality assessment of generated reports but also provides more effective guidance for research and model development in this field. Meanwhile, this is a very challenging task. We cannot hope to get a 100% correlated scoring system as humans in a single shot, but this research has to start from somewhere and we have demonstrated promising solutions.

Questions:

1. Please note that the dataset of 50 samples is only used for the sanity-check of the training data; the validation of our model is conducted on two datasets with distinct criteria, comparable (indeed more than) to the scale of the existing works in this domain.

2. Each predicted report and its corresponding ground truth report must be combined as input for the evaluation process, as all evaluation metrics require both. The ground truth report serves as the reference for assessing the accuracy and quality of the predicted report.

3. The Rad-100 dataset has unique 100 ground truth reports.

4. The "Total" here represents the correlation of the report's overall score between the metrics and the human-rated total score. In contrast, the "Individual" scores indicate the correlation between each specific criterion’s score from the model and the corresponding criterion score given by humans. While the individual scores themselves are summed to produce the total score, the correlations (Total and Individual) do not have an additive relationship—only the scores do.

5. Yes, for table 2, all other NLG metrics need both generated report and ground truth report to calculate the score

Thank you for the feedback

There are no such datasets publicly available. We would like to emphasize the scale of the dataset involved in our work is at the same scale of relevant works in the literature.

Different from existing methods directly predicting the overall score only, our method predicts the subscores and adds them to calculate the overall score. This indeed improves the model generalization capacity, reducing the risk of overfitting to data, because it is a more challenging task to get both the subscores and the overall score to be highly correlated to human ratings. Meanwhile, using our current training dataset, our model has significantly outperformed existing evaluation metrics in correlating to human ratings.

Please note that we used zero-shot prompting to generate the training samples. It is reasonable to expect that incorporating few-shot prompts could further enhance our model's performance, as generally agreed in this field.

4. Please note that the test dataset ReXVal involves 6 human raters and the ground-truth scores are based on inter-human agreement.

Questions.

1. Please note that the ground-truth reports are taken from MIMIC-CXR, while the quality of the reports is controlled by the dataset. We use the same evaluation datasets as existing comparable methods in this domain.

2. About computational cost: The conventional NLG metrics and the existing clinic-scores do not need training, however, they correlate poorly with human ratings and they have no ways to customize to different evaluation criteria. These are the challenges that this work wants to address. Compared with other trainable evaluation metrics like GREEN, our model is computationally very efficient: GREEN’s inference time is 1.06 seconds per sample, while ours is only 0.04 seconds per sample. GREEN has a much higher training cost: it was trained on 8x NVIDIA A100 Tensor Core GPUs with 40GB VRAM each, using a batch size of 2,048 for 12 epochs. In contrast, our model was trained on a single A6000 GPU with 40GB VRAM, using a batch size of 6 for 4 epochs.

3. Our training loss curve converges on the two tasks.

Thanks for your feedback Please note that, among your suggested methods, FineRadScore and G-Rad are not formally published methods and they are not directly comparable to our method, as they rely on online LLMs, which raises privacy issues and is inviable for practical use. In contrast, our model is an offline model.

Compared with GREEN, our method has multiple advantages. First, our Kendall’s tau score is higher than GREEN’s (Ours: 0.751, GREEN: 0.640). GREEN adds interpretability through free-text analysis but sacrifices scoring accuracy due to the task complexity. Furthermore, GREEN’s fine-tuning of LLMs lacks a dedicated loss function as ours, limiting its sensitivity to nuanced quality differences. Second, our inference speed is significantly faster. GREEN’s inference time is 1.06 seconds per sample, while ours is only 0.04 seconds per sample. Third, GREEN has a much higher training cost: it was trained on 8x NVIDIA A100 Tensor Core GPUs with 40GB VRAM each, using a batch size of 2,048 for 12 epochs. In contrast, our model was trained on a single A6000 GPU with 40GB VRAM, using a batch size of 6 for 4 epochs.

As mentioned above, FineRadScore is not comparable to our model, as it only leverages the capabilities of large models (GPT4 and Claude-3 ) without fine-tuning, and scores each report by comparing sentences individually. Notably, it cannot provide an offline model to cater for the privacy concerns for evaluation.

As mentioned above, similar to FineRadScore, G-Rad relies solely on an online GPT model as its evaluator, which is both costly and requires online access.

Thank you for the feedback, 1. First, we’d like to emphasize that the dataset scale in our work is comparable to that of comparable studies in the literature. Evaluating the scoring system's correlation with human ratings requires ground-truth scores from radiologists, which naturally limits the size of the test datasets. Nevertheless, we have evaluated two datasets, including the public ReXVal dataset, using two different scoring criteria, a notable advancement over existing works that rely solely on ReXVal and test only one criterion (e.g., RadCLiQ or MRScore). Could you please provide any other examples that cover more criteria or datasets than ours?

2. Please note that RadCLiQ and MRScore are currently the only available scoring systems explicitly provided in the literature of relevant works. Also note that, no existing methods have evaluated both scoring systems as we did. If additional systems were available, we would apply our method to them as well. By using two distinct scoring systems, we show that our model can be easily customized to various evaluation criteria.

While our model requires retraining to adapt to new scoring systems, this represents a big leap compared to existing NLG metrics and clinical scores, which have NO WAY to adapt to different evaluation standards.

In Table 1, although the correlation values vary across different sub-criteria, they are all statistically significant, and the overall score, which is the addition of all sub-scores, shows very high correlation. The variation of correlation values may be related to the distribution of training samples according to the sub-criteria. For example, in MIMIC-CXR, there are only a small amount of reports that have clinic history records, which may contribute to the relatively low correlation of the “omission of comparison that notes a change from a previous study.

Please note that the margin is NOT a hyperparameter, but was calculated using the difference in the scores of the training pairs (i.e., accepted and rejected reports) generated by GPT-4 . If we do not use margin (i.e., set margin = 0), the Kendall-tau correlation will drop from 0.751 to 0.731 and the Spearman correlation will drop from 0.910 to 0.895.

Thank you for your feedback. To address your concerns, we have outlined the following points:

1. Our model is significantly different from RadCliQ and MRsore.

First, neither method can conduct fine-grained evaluation, e.g., by combining sub-criteria, we can clearly identify the reasons for a report’s poor quality, e.g., whether due to incorrect lesion location, incorrect severity of findings, or omission of findings, which is not achievable by either RadCliQ or MRScore. Notably, simultaneously predicting subscores accurately is a more challenging task requiring a carefully designed optimization process. In contrast, RadCliQ combines non-trainable scores linearly, offering limited improvement and flexibility, while MRScore can only predict an overall score. Even for the overall score, we improved the Kendall's Tau from 0.63 (the second best) to 0.75 and Spearsman correlation from 0.81 (the second best) to 0.91 on the public ReXVal dataset. That is a significant improvement in any means.

Secondly, we design completely different loss functions from existing LLM-based scoring systems, beyond enabling subscore evaluation.

Thirdly, it is the first time that an LLM-based R2Gen scoring system can demonstrate its adaptability across different evaluation criteria.

Moreover, as for "the customization feature highlighted as a key contribution lacks sufficient experimental support, with limited results in Table 2 that do not fully demonstrate its claimed flexibility", please note that Table 2 alone is not used to demonstrate the flexibility of customization feature!!! The customization feature is demonstrated by our experiments on two datasets with different scoring systems (results presented in Table 2 and Table 4).

2. To score existing R2Gen models, we need to access the reports generated by these methods, which are absent in most cases. Also, although this is an interesting work, it totally goes beyond the scope of this paper, which focuses on designing a new evaluation method and comprehensively evaluating its performance.