Century: A Framework and Dataset for Evaluating Historical Contextualisation of Sensitive Images

We thank the reviewer for their thoughtful comments on our manuscript.

We note the reviewer’s concern on the dataset scale. We designed the dataset for use in evaluations of historical contextualisation capabilities of multi-modal models - not for model training or fine-tuning. With this dataset, developers will be able to assess how well models contextualise sensitive historical images. They can use Century to inform post-training and deployment decisions While we invite future work on improving system capabilities, the dataset does not include "target" responses of how systems responded to the different classes of queries for the images in this dataset (ie. what is being evaluated in Table 3). We only release the images and related metadata - a point we make clearer in the updated draft (lines 320-323). We do hope that the dataset enables application and system developers to make their own ethical and normative choices in their deployment context (possibly adapting or building on our work).

In constructing our evaluation task, we developed several criteria along which we test for the “goodness” of the historical contextualisation response. Surprisingly, even though models are almost certainly exposed to images and their contexts in training, they perform poorly even on fairly objective elements of the task such as identification. This can be clearly seen in the accuracy results we provided in Table 3 of our manuscript where we showed that the best model achieved 53.3% accuracy on this task. Because of this and the other empirical results in Table 3, we don't think training on “target” responses would be likely to lead to improvements along even the accuracy dimensions of our criteria, let alone others that we evaluated in this work.

We cite evaluation frameworks related to historical contextualisation from images in the Related Works section, but to our knowledge, Century is the first benchmark focused on evaluating multi-modal generative models on historical contextualization of images. We have added the two suggested citations on automated metadata annotation and evaluating visual reasoning in context-rich scenarios in our discussion of how Century builds on previous efforts to measure visual reasoning and contextualisation (line 473, 475).

We understand the reviewer’s concerns on the potential bias in the benchmark. The most thorough analyses of potential biases in Century can be found in Table 8 in Appendix K, where we look at the distribution of figures, events, and locations represented in the dataset by UN world subregion. We find that as per the human evaluation labels, every subregion is represented by at least 5.1% of the images in Century, indicating that our dataset contains a relevant amount of images from all areas of the world. Unsurprisingly, we also find that images taken from Wikipedia are skewed in their distribution (with images from Western Europe and North Americas most prominently featured), but we do find evidence that Century contains images from all areas of the world.

We make recommendations on mitigation strategies that could be applied in future work in the Limitations, calling upon researchers to integrate participatory perspectives and build upon the representativeness of the work in a targeted way (e.g. contributing images of a specific culture). We also are excited to see future work build on our methods for creating the dataset, and adapt them to other historical data sources (eg, cultural archives).

Regarding the fit of this paper in ICLR, we are responding to the call for papers, which explicitly seeks work related to "generative models" and "datasets and benchmarks" and "societal considerations including fairness, safety, privacy." Our work directly speaks to multiple subject areas in the CFP. Additionally, previously accepted papers in ICLR 2023 and 2024 include papers that center contributions through datasets and benchmarks:

• MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations, ICLR 2024
• SWE-Bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024 (oral)
• MEDFAIR: Benchmarking Fairness for Medical Imaging, ICLR 2023 (notable-top-25%)

We thank the reviewer for their careful engagement with the paper and their suggestions to strengthen the manuscript, which we have included in the updated version. If we have addressed the reviewer’s most pressing concerns, we kindly ask the reviewer to consider adjusting their score to reflect this. Otherwise, we are looking forward to proceeding with the discussion and incorporating any further feedback to our manuscript.

We appreciate the reviewer's thoughtful feedback on our manuscript. We're pleased they recognised the paper's comprehensive approach to addressing a significant evaluation challenge using a diverse dataset.

Regarding the reviewer's suggestion to evaluate open-source models, we agree that this would be valuable for the community. However, most open-source multimodal models (like PaliGEMMA) require substantial fine-tuning to handle complex tasks like historical contextualization. The few systems that are fit for this purpose are not available for use due to their licenses, so we were unable to evaluate these systems directly. To encourage open investigation, we've made our dataset and evaluation methods publicly available, enabling researchers to explore how current and future systems (including new tokenization strategies, inference-time compute methods, and composite systems) impact historical contextualisation capabilities.

As recommended, we've expanded the limitations section (lines 502-506; 513-517; 524-530) to address potential unintended consequences of our methodology. We also discuss short-term mitigations for developers to avoid potential pitfalls when optimizing for Century, alongside longer-term improvements.

We've added clarification to the captions of Figures 6 (page 23) and 11 (page 34) to improve their interpretation, addressing the reviewer's concerns about the figures.

Regarding demographic data for raters, we are required to store rater data with obfuscated identifiers, which means demographic data for individual raters are not easily recoverable. We acknowledge this as an important addition to future work, especially participatory work that seeks to identify the perspectives of a specific group of people on the historical contextualisation task. We provide details on our the recruitment strategy decisions that likely influenced ater pool composition in lines 305-307 and Appendices L and T.

We thank the reviewer for their careful review and insightful comments, which have strengthened our manuscript.

We thank the reviewer for thoughtful comments that scrutinise how historical contextualisation – a higher-level capability our benchmark claims to measure – may consist of multiple lower-level fundamental capabilities, including but not limited to entity recognition, scene understanding, and world knowledge.

How different capabilities may be expressed when an AI system performs a complex task, such as historical contextualisation, is an important research question. We note that the reviewer proposed unpacking into low-level tasks is not the only possible approach. In order to output a contextualised description of a historical image, such as a photograph from World War 2, a given model may be invoking any number of latent capabilities. Which capabilities are expressed and how they are encoded is likely to vary across model specifications, such as its technical architecture, training procedure and multimodal corpora it is trained on.

Measuring specific lower-level capabilities has been the focus of many previous work (for example, MS-Celeb-1M for entity recognition, OKVQA for scene understanding, InfoSeek for world knowledge, Dollarstreet for fairness in object recognition, TallyVQA for object counting, DocVQA for OCR in context). While there is established coverage for low-level vision tasks, we identified a gap in the available evaluations that inspired us to create Century: there was no measurement for how well multi-modal models could contextualise and reason about images grounded in real-life events in open-ended text generation tasks.

While our focus has been on measuring the higher-level historical contextualisation capability, we hope that releasing Century along with an evaluation protocol will enable additional studies of the interplay between low-level and high-level capabilities as pointed out by the reviewer.

In this work, we propose a first decomposition which partially overlaps with the proposed (1) identification (2) association (3) fair / objective description breakdown.

We target Capability (1), the identification of the event or person depicted in the image, in our evaluation method as the “correct identification” question (Table 3). We ask auto-raters and human raters to identify if the target model correctly names the entity depicted in the image, providing a measure of the “perceptual capability” of the model.

For (2), the reviewer makes an interesting point about the capabilities that may be necessary to associate figures and events depicted to their historical context. Qualitatively, we find that different images are more difficult to contextualise given the image alone (e.g. identifying the context behind a photograph of a famous political figure is more straight-forward than associating an image of a crowd with a specific historical event). We discuss this point in the “Recommendation for Developers” section, but have pulled a longer discussion into results section to provide guidance on interpreting results in light of the differences in contextualisation difficulty (lines 425 - 473).

For (3), or the “goodness” of the generation, this is covered by several of the evaluation criteria, including “factuality,” which evaluates if inaccuracies or factual errors are present in model output, and “appropriate summary,” which evaluates how much relevant detail on the event depicted is present in the model output.

These partial overlaps in our decomposition approach and the approach recommended by the reviewer may indicate that the two approaches are complementary.

To the reviewers’ concern on the paper not fully describing dataset contents, we have added content in Section 3 to clearly state what fields are included in the datasets we are open-sourcing (lines 320-323). Alongside the publication, we will link the Github page, which will also describe the different fields contained in the dataset.

We thank the reviewer for their careful engagement with the paper and their suggestions to strengthen the manuscript, which we have included in the updated version. If we have addressed the reviewer’s most pressing concerns, we kindly ask the reviewer to consider adjusting their score to reflect this. Otherwise, we look forward to proceeding with the discussion and incorporating any further feedback to our manuscript.

We thank the reviewer for their thoughtful review and appreciation of our contribution. We are pleased the reviewer recognizes the value of Century in addressing a key evaluation gap.

Regarding the reviewer's concern about potential biases, we acknowledge that platforms like Wikipedia, from which we draw data, may introduce bias. However, our analysis (Table 8 in Appendix K) shows that Century includes images from all UN world subregions, with each represented by at least 5.1% of the total images. While there is an over-representation of Western Europe and North America, likely inherited from Wikipedia, Century still demonstrates a global reach.

Initially, we intended to pursue direct collaboration with cultural heritage institutions and archives to increase representation for certain groups, but later found this was not feasible for this project. However, we believe Century provides a strong foundation for future partnerships and research with these institutions.

We agree that incorporating participatory methods to assess image sensitivity is crucial, and we have highlighted this direction in our discussion under "Lack of targeted inclusion of affected communities." By releasing Century with geographical labels for each image, we hope to facilitate future research that includes participatory perspectives, including ethnographic studies focused on under-represented populations.

In response to the reviewer's questions:

• Automating Century evals: We demonstrate the performance of off-the-shelf foundation models as raters for the historical contextualization task. Table 3 (pg 3) demonstrates the directional alignment between the human ratings and autorater decisions. However, we believe more work is necessary to increase the accuracy and calibration of automated evaluation, and recommend retaining human evaluation for final model comparisons given the sensitivity and subjectivity of the task.

• Potential misuse of the dataset: We believe the risk of misuse is low. We do not release any raw model outputs or human/auto-rater signals that could be used to train models to generate harmful outputs. Section 3 clarifies the specific fields included in the open-sourced datasets (lines 320-323).

• Exacerbating biases through training: Century is not intended as a training dataset, so we do not anticipate it contributing to bias amplification in that way. However, we recognize the importance of improving dataset representativeness to enable effective evaluation of model performance across diverse communities.

We thank the reviewer again for their thorough review and appreciate their view that this work offers a compelling contribution to the ICLR community.