Alt-Text with Context: Improving Accessibility for Images on Twitter

Thank you for your time and feedback! It’s true as we note that the alt-text found on Twitter is largely written by untrained users. We find that in most cases it is functional, but we did want to at least acknowledge that there is some noise in our data, as with any user-generated corpus. To some extent this is an inherent problem with collecting a large-scale alt-text dataset in the first place. If we attempted to crowdsource the alt-text we would face similar problems, and hiring a small number of experts would severely limit scale. And despite some unavoidable noisiness, our dataset is the first of its kind at this size for this task – there weren’t really alternatives to work on instead. We also hope that to some extent this averages out with scale, i.e. over a large enough dataset the underlying signal will be learnable and measurable.

Furthermore this is why in addition to computing automatic metrics comparing model outputs to the gold references, we also perform human evaluation (see below) where we ask sighted users to compare generated alt-text from various systems. The original user-written captions do not factor into this experiment, but we see similar results to our experiments using automatic metrics.

Regarding BLEU score, we are using a publicly available implementation of all automatic metrics from pycocoevalcap, the same one used by ClipCap and several other papers. As indicated in the Table 1 caption, all numbers are x100 (i.e. BLEU ranges 0-100), which is standard practice for reporting in much of the prior literature.

We tried to follow ICLR’s instructions for supplementary material by uploading it separately from the main submission PDF. The appendix is in its own PDF wrapped in a zip file, which you should see listed under “Supplementary Material”. At least from our end it can be downloaded and viewed from the OpenReview submission page. Please let us and the chairs know if you still cannot access it.

Regarding rehydration, we understand your concerns and wish that there was a more convenient way to release our dataset as we do believe in the importance of our research being replicable and easy to follow up on. However we have to balance this with users’ right to be forgotten and the website’s terms of use. If a user no longer wishes for their image and associated tweet to be either online or in an open training set and deletes them, we have a responsibility to comply with that. We’re in a tough situation here as the API was changed while this paper was in progress, but we believe that this approach has the best tradeoff between facilitating reproducibility and respecting user rights. If you have other suggestions for how we could do this, we’re certainly open to hearing them.

The human evaluation was conducted in a systematic way – there’s some information in 5.2 and more in Appendix E (see above). To summarize, we randomly selected 1k examples from our test set and hired annotators from amazon mechanical turk to assess model outputs. Turkers were given detailed instructions on the task, shown contrasting examples of both well and poorly written alt-text, and asked to read a document with further details on best practices according to Twitter’s accessibility guidelines. Before allowing them to participate in the actual study, we also conducted a dummy round on a small number of tweets with two candidate alt-text descriptions each. Only turkers who correctly selected the best alt-text for all of these were allowed into the next round which we report numbers on in our paper.

We consulted with legal support and arrived at the conclusion that for the purposes of this study, the turkers would not be regarded as human subjects, but rather as annotators. This is because we were not observing or gathering personal information from or about them, but rather hiring them to evaluate our data. Based on this we were under the impression that this assessment would be IRB exempt.

We have an ethics section as well as a limitations section in our appendix (see above). We’re sorry those were hard to find. We do definitely agree that there are important ethical considerations with our research, especially with regard to the potential impact on quality of user experience for BLV people. We did also take steps as described to filter out personal information such as names from our corpus. With regard to annotators viewing harmful information, we are to some extent dependent on Twitter’s ability to enforce its own terms of service, and dynamically populated tweets and images such that anything that had been flagged for content in the several months between initial collection and the final round of human eval would not render. Finally we did also provide both a warning and mandatory acknowledgement from annotators before starting that they might encounter NSFW content, and also indicated this in the title of the task listing.

Thank you for your time and feedback! We’d argue that the tweet text reranking is far from the most novel thing in our model (as well as not being the primary source of our quantitative gains), rather the way we approach alt-text generation (which is in general understudied) by way of contextual text conditional image captioning is in our opinion more new and interesting, and also accounts for more of our numerical performance improvement over prior work. While work exists that has projected image features into text space, there’s much less precedent for our way of creating a prefix from multiple modalities, which we empirically demonstrate is what’s responsible for our gains.

On the topic of reranking though, we’d like to clarify our experiments – and critically emphasize that as stated in our paper, we did try reranking on ClipCap and found that it yielded similar results. We’ll try to make that more clear in the next revision. Specifically we found that it improved performance there as well by a similar amount, for example boosting BLEU from 0.681 to 0.793 (we’ll include those numbers in the next revision as well). You’ll notice by comparing the two bolded rows in the table that improvement from reranking itself is similarly small but noticeable for our model also. But we want to strongly emphasize that outside of the ablation at the bottom section of the table, the comparisons between our model and the baselines are apples-to-apples as far as the decoding strategy and therefore fair to directly compare. Column 2 indicates the decoding strategy used for each row, and you can see rows corresponding to our model and ClipCap decoded with the exact same strategy (i.e. no reranking for either). In fact the primary gain comes not from reranking but from the specific combination of beam search with no repeats which is exactly what we do for ClipCap in all experiments even though they themselves in their original paper do not.

We did also try various other reranking strategies, including reranking with BLEU instead of ROGUE, reranking with an inverse model that predicts the tweet text from the alt-text, and reranking by the CLIP text embedding similarity as you suggested. These all do worse than ROGUE, but better than not reranking at all. The CLIP text embedding similarity reranker for example gets 1.834 on BLEU. We can add those numbers to the next revision as well.

Thank you for your time and feedback! We apologize that the definition of alt-text wasn’t made more clear in the paper. Alt-text does in fact have a fairly specific meaning, it’s a well established accessibility feature with generally agreed upon best practices (https://help.twitter.com/en/using-x/write-image-descriptions). Alt-text is meant to be a literal image description that unlike a traditional caption is both more objective and focuses on contextually salient details which may vary with the context in which the image appears (hence our motivation for conditioning on surrounding tweet text). To answer your question, alt-text should ideally not be written very differently from person to person as it is meant to be an objective description that can stand in for the image itself.

It’s true as we note that the alt-text found on Twitter is largely written by untrained users. We find that in most cases it is functional, but we did want to at least acknowledge that there is some noise in our data, as with any user-generated corpus. To some extent this is an inherent problem with collecting a large-scale alt-text dataset in the first place. If we attempted to crowdsource the alt-text we would face similar problems, and hiring a small number of experts would severely limit scale. And despite some unavoidable noisiness, our dataset is the first of its kind at this size for this task– there weren’t really alternatives to work on instead. We also hope that to some extent this averages out with scale, i.e. over a large enough dataset the underlying signal will be learnable.

Furthermore this is why in addition to computing automatic metrics comparing model outputs to the gold references, we also perform human evaluation where we ask sighted users to compare generated alt-text from various systems. The original user-written captions do not factor into this experiment, but we see similar results to our experiments using automatic metrics. To answer your question, we also performed fairly thorough worker vetting for our human evaluation as described in our Appendix, in order to ensure consistency in judgements. We could potentially perform a followup study to more directly measure the extent of inter annotator agreement for the next revision.