PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Models

I understand that it is appealing to call a model "PAPER CLIP" but this name is misleading

We note that we do not call the model PAPERCLIP; instead, as emphasized in the abstract and elsewhere, PAPERCLIP refers to the method of using (processed) proposal abstracts to inform observation-text associations for astronomical images, which in this specific case we apply to observations from the Hubble Space Telescope.

The dataset used in this study is an exception in astronomical data, since images have meaningful textual counterparts. The most crucial question is whether it is possible to use the model trained on the Hubble proposal dataset to label/search some other available images.

The dataset includes diverse images from Hubble, with different filters, targets, and post-processings, taken over a ~30 year timespan. Our method is intended to be applied to observations from a specific telescope, and we do not expect it to immediately generalize to others without further fine-tuning. We emphasize that not all Hubble images have associated text i.e., not all are the downstream result of targeted proposals, as those used in fine-tuning here (the data used is a subset of Hubble observations). Our method is therefore intended to be applied to unlabeled sets of observations. In the paper, in order to be able to compare with a ground truth, we tested our method on observations which do have associated abstracts.

[...] why image-text association should be noisy [...] InfoNCE allows for multiple positive pairs, so it is possible to train model taking all image-text correspondences into account simultaneously.

Although the version of InfoNCE used in the paper allows for a batch of $N$ positive pairs, each element of the pair has $N-1$ negatively paired elements; a single data point cannot have multiple positive “anchors”. Therefore, since multiple images can correspond to an abstract, when training with minibatches it is possible that some of the negative pairs will actually be positively associated (e.g., if two abstracts in a batch are the same, but the corresponding images are different). Through the results in the paper, we empirically show that, despite this, the model is able to leverage the signal between positively associated pairs and “beat down” the noise associated with negatively pairing image-text samples that may nevertheless be associated.

[...] did the authors have a separate test set?

Due to practical limitations associated with the small size of the fine-tuning dataset, we did not use different datasets for validation and testing, deeming this sufficient for a proof-of-principle exposition.

The images are cropped to the 20% of the area of the original image, would this somehow get rid of informative parts in an image?

The images are cropped to 20% on the fly during training, as a form of data augmentation. Since each image is seen ~20 times over training, each time cropped differently, the model can learn to leverage a positive association signal when this is present. The association signal may also not be localized to a particular part of the image.

Why is the random expectation of retrieval accuracy 10%? In 3.3, the formula has the k% of the number of images, but the explanation mentions top k% of captions [...]

The random retrieval accuracy is 10% because, without a learned association between modalities, the model will correctly retrieve the correct item in the top-k%, k% of the time by random chance.

As mentioned in Sec. 3.3, the retrieval evaluation metric when averaged over the test set is symmetric over the image and text modalities; i.e., due to commutativity of the cosine similarity, we can either average over images in the test set while looking at top-k% captions, or vice-versa. In Sec. 3.3, for concreteness we describe text retrieval (“top k% of captions”, “averaged over the images”). Fig. 2 right therefore does not correspond to a particular retrieval task.

[...] I was not sure what the authors meant by “it is challenging to discern meaningful, strong associations”

While occasional associations are apparent, this is very much the exception rather than the norm for the base model (as can be seen from subsequent retrieved images); however, we agree that the language here should be softened.

[...] the authors claim that the fine-tuned model “shows strikingly different behavior’’, which, as a non-expert for the astronomy domain, was not clear to me.

The difference in behavior can be seen from the retrieved images – in Tab. 4 all the returned images for the fine-tuned model correspond to Jupiter and SN1987A, as appropriate. This is not the case for the base model, in Tab. 3. While domain expertise may be helpful in assessing relevance, one can click through to the Proposal IDs of the retrieved images, confirming their relation to the queries.

There is no technical novelty in data construction or modeling. The dataset is constructed by downloading abstracts and matching with images. The model is standard CLIP. The experiments on the constructed validation set are as expected.

Only a loose connection with LLMs. The paper is primarily about CLIP.

We frame our study as an application of language models to a previously-unexplored field with domain-informed data curation. We note that the COLM call mentions, “LMs on diverse modalities and novel applications [...] with extra encouragement for less studied modalities or applications.”

Regarding the paper primarily using CLIP and having a loose connection with LLMs, we again note that call for papers mentions, “We consider the term "language model" in the broadest way.”, and is thus not restricted to LLMs specifically.

Overall, we believe that an LM application in astronomy, which is an understudied field in the context of LMs with nevertheless high potential, is in-scope for COLM.

The paper does not include any baseline comparisons. [...] 1), Are there existing tools capable of performing similar tasks? 2) Assuming each observation has an abstract, could description/image retrieval be achieved by calculating text similarities [...]

This is a great question. The reason the target application is not possible with other methods (e.g., text similarity) is that not all Hubble images come with descriptive captions, like those used for fine-tuning the CLIP model (i.e., our observations are a small subset of all Hubble observations). Large survey-style campaigns image different parts of the sky without target-specific captions, which is the setting where the current method would be useful to query for phenomena of interest in large unlabeled datasets.

[...] is there some way to see how alinged the astronomical data with the CLIP model's domain? It is unclear whether the observed performance improvements are due to fine-tuning (from a generalist vision model) or if the original CLIP model had prior exposure to similar astronomical data, enabling rapid adaptation.

While this is difficult to check for the proprietary CLIP model used in our study, it should be possible to check for similarity of image embeddings in public datasets like LAION-5B. Unfortunately, these datasets are currently (May 2024) inaccessible due to safety concerns. However, given the domain-specialized nature of our data, it is highly unlikely that images of this kind were included in the CLIP trained set. The poor performance (Fig. 2 right) and lack of diversity of cosine similarities (Fig. 3 left) in the base model also indicate this. The fact that training from scratch does not match the performance of using the pre-trained model further indicates that a generalist backbone is necessary for performance.

Q:The paper extensively discusses the use of LLM-generated abstracts, though this approach does not yield superior results. Can the authors delve deeper into why this might be?

The data-processing inequality implies that the summarization process is necessarily lossy; the fact that the raw abstracts perform quite well implies that the CLIP training procedure is able to extract relevant information from the abstracts dataset, even though it is noisy. This capability was not a-priori expected, and is a central message of the study.

The need for PAPERCLIP needs to be better motivated in the paper.

Traditionally, astronomers train specialized models to search for phenomena of interest. This requires the curation of training data and separate models for each application. The motivation for this work is to introduce a method to train a single model that can be used to query relevant images from a set of observations, based on natural language.

Shouldn’t the quantitative experiments be designed to test image retrieval [...]? [...] one might query for images [...] It would be great if the authors extended this setting to a formal benchmark for quantitative evaluation.

The retrieval accuracy, as defined in Sec. 3.3, is precisely the image retrieval accuracy for an image-text pair validation set. Note that this metric is symmetric in text to/from image retrieval, when averaged over the entire validation set.

[...] the models trained with abstract should also be tested on the validation set containing the summaries.

We do exactly this in Fig. 3 (right) – here, all models (including trained with abstracts) are tested on the summaries, for a direct comparison.

Will the model perform well on other datasets not included in the training? Table 1 has two images: one from 1999 and the other from 2016. There are some obvious quality differences. Did you notice if these differences propagate to the model?

The dataset includes diverse images from Hubble, with different filters, targets, and post-processings. The differences between the images in Tab. 1 is coming from these variations, rather than differences in observation years. Our method is intended to be applied to a diversity of observations from a specific telescope, and we do not expect it to immediately generalize to different telescopes without further fine-tuning.

The CLIP text encoder has a maximum text length of 77 tokens. What do you do when the abstract length is greater than 77 tokens? Do you discard it?

We note in Sec. 3.2: “When using raw proposal abstracts, random chunks of the text delimited by periods are selected on the fly to fit within the maximum token length of the text encoder.” This also serves as a form of text data augmentation, given the relatively small dataset.

We emphasize that for scientific usefulness, the goal is not necessarily to
correctly retrieve the most relevant objects, but rather to identify a diverse set of interesting
candidates for manual follow-up and further analysis.

@outlines.prompt
def prompt_fn(abstract, query):
     """[INST]
You are an expert astrophysicist, with broad expertise across observational and theoretical astrophysics.

Abstract: "{{abstract}}"
Query: "{{query}}"

The above is an abstract for a proposed observation taken by the Hubble Space Telescope (labeled "Abstract"), and an object or concept (labeled "Query").

Could the observations corresponding to the abstract contain the query? Be precise, and do not contain related concepts or objects. 

Your response should be either True or False. Only return True if the query is closely related to the abstract, and the downstream observation could be relevant to the query.
[/INST]
"""