ContextRef: Evaluating Referenceless Metrics for Image Description Generation

We thank the reviewer for their thoughtful engagement with our paper and for raising the following important point, which we have now addressed in the paper.

> This work involves human ratings, which are subjective and can introduce bias. The paper could benefit from a more detailed discussion of how raters were selected and trained to ensure consistency and mitigate potential biases.

To address this point, we added a subsection with details on the participant population (Appendix A) and we added a detailed limitations section (Appendix I). For convenience, we summarize our response here, as well.

The goal of the ContextRef framework is to test model-based metric alignment with people’s intuitions for description quality for an accessibility goal. However, as the reviewer rightfully points out, there is interannotator variance, and people’s cultural backgrounds and individual needs necessarily shape their preferences. To get first traction on the core problem itself, we get multiple ratings for each individual description, which reduces the effect of outliers but reflects general preference patterns in the recruited population. This is not to say that outliers aren’t meaningful, and we added a discussion on two core sources of variance.

Firstly, we elaborate on the annotator selection and discuss the implications of it. This includes that recruitment was restricted to US-based adults who are part of the Prolific platform. We agree that this leads to implicit biases in the data that might not generalize to participant populations outside of our recruitment parameters, which is a clear limitation that we now elaborate on in the paper. In the paper, we focus on a pipeline that allows us to get initial traction on the overall problem of using models for image description quality assessment, but this doesn’t yet allow insights on important perspectives such as cultural representation and equity promoted by these models, which need to complement our framework.

Secondly and in addition to cultural variation, there is individual variation, which is an important component of current image description research. One such example is description length. A variety of papers have found conflicting evidence on whether shorter or longer descriptions are generally preferred (Slatin & Rush, 2003; Petrie et al., 2005; Rodríguez Vázquez et al., 2014). We therefore chose to set up this study based on a goal-oriented framing (consider the purpose and choose your responses accordingly) as opposed to explicit training and instructions. However, we encourage work that builds on ContextRef to adjust the experimental design according to potentially distinct downstream needs. To incorporate the individual uncertainty into our framework, we correlate metric predictions to individual rating distributions instead of the averaged ratings for each description.

Finally, we would like to emphasize that we chose this experimental method since it was established by prior work with sighted as well as Blind and Low-Vision (BLV) participants, which helps ground our findings in the existing literature.

We hope that this addresses the reviewer’s concerns but if there are any particular points, we can expand on, please let us know.

–

John M Slatin and Sharron Rush. Maximum accessibility: Making your web site more usable for everyone. Addison-Wesley Professional, 2003.

Helen Petrie, Chandra Harrison, and Sundeep Dev. Describing images on the web: a survey of current practice and prospects for the future. Proceedings of Human Computer Interaction International (HCII), 71(2), 2005.

Silvia Rodríguez Vázquez, A Bolfing, and P Bouillon. Applying accessibility-oriented controlled language (cl) rules to improve appropriateness of text alternatives for images: an exploratory study. Proceedings of LREC 2014, 2014.

Thank you for the encouraging comments and supportive feedback! Indeed, the relatively poor performance of these models in scoring image descriptions in context did not come as a significant surprise to us – prior work showing the poor performance of referenceless metrics in this setting (e.g. [1]) was a significant motivation for this work. Our paper establishes a solid empirical basis for this claim, offers new evidence that context is crucial, and points the way to variants of these metrics that are more successful, with new data and new fine-tuning results.

At the same time, one may find the poor performance of likelihood-based models somewhat surprising because, indeed, one may argue that these models are specifically trained with context. In particular, language models have been consistently shown to extract remarkably much from information simply presented in their context windows; this property of large pretrained language models has been the subject of extensive interest in NLP [2,3].

Thus, we’d like to emphasize that, 1) while the observation that these models perform poorly in context is in line with prior work, the universality of this property was likely not something we could have anticipated a priori and 2) we expect there may be additional strategies that leverage pretrained models in novel ways to better incorporate contextual information into existing referenceless metrics, as done in the contextual CLIPScore from [1] – in fact, this is one of the motivations for creating this benchmark.

Finally, independent of how surprising the poor performance of models for integrating context might be, our goal is that this framework will help to push the development of metrics that are sensitive to image context and therefore lead to improvements in the theoretical approach to the problem and, eventually, the downstream usefulness of those systems.

Based on your comment, we further highlighted the context integration procedures in Section 3 and Appendix H, as well as in the extended related works discussion in Appendix J.

--

[1] “Context matters for image descriptions for accessibility: Challenges for referenceless evaluation metrics,” Kreiss et al. 2022

[2] “An Explanation of In-context Learning as Implicit Bayesian Inference” Xie et al. 2021

[3] “Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?” Min et al. 2022

We thank the reviewer for all of the constructive comments and helpful suggestions. We updated the paper accordingly, and summarize our changes and responses here. We hope that they address the reviewer's concerns; however, if there are specific areas where we can further improve, please don't hesitate to let us know.

> The related work section is focusing almost solely on the clip model,and fails to provide an accurate review of pre-existing related literature. Authors should review pre-existing metrics, related benchmarks and evaluation strategies.

We agree that the paper benefits from discussing those related works in more detail and updated our paper accordingly. Since we had to contend with the space limitations, we opted for discussing much of the related literature in the thematically corresponding sections wherever possible (Section 2 for referenceless metrics and robustness checks, Section 3 for relevant models, Section 4 for evaluation frameworks and annotation guides, and Section 5 for common model-generated errors) and specifically elaborate on pre-existing metrics as well as prior evaluation strategies and benchmarks in a new Appendix section (Appendix I).

> Section 3 provides descriptions of metrics being analysed in this work, and lacks accurate descriptions to understand the particularities of each method. For example, only encoder architectures are discussed for the Flamingo model, but the actual mechanisms used to compute the metric are not discussed. The strategy used to integrate context information is also poorly explained, and could benefit from providing equations.

Thanks for pointing this out! We’ve added some equations to clarify how context is incorporated, we streamlined the model details provided in Section 3 and added more model details in Appendix H. For the likelihood-based approaches, context integration is simply accomplished by including the context as part of the input text when computing likelihood. There are certainly other ways of incorporating context into these likelihood-based models, but we leave this to future work. We’ve also reproduced the equation from Kreiss et al. (2022) to help clarify the role of context in the tested CLIPScore.

> Authors mention that their results suggests that referenceless metrics can be successfully used based on observed correlation, however somewhat low correlations are observed (.2-.3), suggesting the opposite.

We agree that the previous version of the paper failed to properly contextualize the size of the correlations and we are aiming to address this here. Due to the interannotator variation, the highest correlation that is theoretically achievable on our dataset is actually much lower than 1. Depending on the question, it ranges between 0.58 and 0.66. (We computed this by correlating our dataset with the average rating for each description.) However, this still highly overestimates the realistic upper bound for a correlation. To address this, we created a counterfactual dataset by sampling with replacement from the original dataset and correlated the average ratings of the counterfactual dataset with the original data. This suggests a more realistic human upper bound with our data at between 0.42 and 0.57 depending on the question. These numbers are further supported by prior work correlating different participant populations on description quality tasks. Kreiss et al. (2022) found a correlation of about 0.5 when comparing sighted and BLV user judgments. These prior observations better contextualize the quality of the correlations reported in the paper. We agree that correlations below 0.3 should not be considered strong and we agree with the reviewer that our paper doesn’t rest on the assumption that those correlations are particularly high. In the updated paper, we adjust our language accordingly. However, for post-image ratings on the overall question, 5 out of the 7 metrics achieve a correlation higher than 0.35 and we believe that the estimates of an upper correlational limit suggest that correlations of 0.35 and above are very impressive, given the interannotator variation that is a core characteristic of psychological preference studies. We also added further discussion on the interannotator variation in Appendix H. We integrated the estimated correlation ceiling into Figure 2 in the updated paper to help readers contextualize the size of the correlation.

(We also realized that the y axis breaks in Figure 2 weren’t well chosen to identify the correlation values. We therefore changed y axis breaks in Figure 2 to make the values clearer and added a table with the exact correlation values to the appendix.)