We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:

It is unclear whether the observations made by the authors are generalizable cross other datasets and modalities.

This point is also mentioned in our limitations (Section 4). Our broad goal is to demonstrate that the cross-modal memorization findings we discovered can emerge, and they are possible to happen in other modalities. Therefore, we hope our work can serve as a starting point for fellow researchers to study memorization in different modalities.

It might have been good to run all experiments on COCO dataset instead of the synthetic dataset and use synthetic dataset as OOD.

Using COCO will be more realistic, but we use fully synthetic data to prevent data contamination. With real images, it is often unclear whether the associated knowledge has already appeared during pretraining, making it difficult to isolate cross-modal memorization effects. Our synthetic setup enables a controlled investigation of memorization without the confounding risk of training data leakage. Additionally, using real human faces might raise serious privacy and consent concerns.

In terms of VLM memorization there is this other paper that is not cited.

Thank you for pointing out this paper on CLIP models. We will include a discussion of it in the camera-ready version.

The paper first mentions that there are ~64B profiles but then later that there are only 100 unique profiles. How did you ensure that the dataset is diverse and representative enough for the study?

The total number of possible attribute combinations is approximately 64 billion, which reflects the size of the design space. We included this to emphasize the diversity and richness of the attribute pool. From this large space, we sample 100 unique profiles to construct a diverse and representative dataset while keeping the experimental setup manageable.

How much data was added during training with variants? How did you know how many variants you need to add?

The number of variants depends on the target number of training epochs. For example, if we aim for 20 epochs, we add 19 additional variants per profile, alongside the original, resulting in 20 distinct samples. Then, we perform a single training epoch, where each variant is seen once—effectively simulating a 20-epoch training schedule with different variants.

We hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions.

We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:

To me, the main weakness of this paper is in the way is it framed.

We understand the concern. While prior work often focuses on unintentional memorization from large-scale pretraining, such experiments are extremely expensive to conduct—especially for VLMs, where training costs are significantly higher than for LLMs due to the length of visual tokens.

Our experimental setting represents a worst-case privacy scenario, where the memorized signals are not diluted by non-canary data. This setup is particularly useful for stress-testing mitigation methods. We will revise our wording in the camera-ready version to improve our clarity.

It's also not quite correct that there has no work at all on multimodal memorization.

Thank you for pointing out these two papers on CLIP models. They are indeed closely related to our work, and we will include a discussion of them in the camera-ready version.

We hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions.

We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:

The paper is very engaging, which made me want to know more about potential causes for the observed patterns, something that wasn't touched upon very much.

Thank you for the positive feedback and your interest in the underlying mechanisms. We agree that understanding the causes of cross-modal asymmetries is a valuable and exciting direction for future work. Thoroughly investigating these causes may require a different set of experiments. We would like to focus on this in our follow-up work.

In a similar vein, I would have loved to see more results on other mitigation strategies, although it was clearly acknowledged in the introduction that this was for future work. Could it not have been one big paper?

We tested out straightforward baseline methods in this paper. However, designing mitigation methods that can truly close the modality gap likely depends on first understanding the underlying causes—as you noted above. This would require a systematic and non-trivial analysis. Rather than overloading a single paper, we felt it was more appropriate to separate these efforts and ensure each component received the depth it deserves. We see this as an important next step and appreciate your encouragement in that direction.

The results are based on one (specialized) dataset and one model family (the focus on solely image-text modalities was already acknowledged in the paper).

We have also run additional experiments with LLaVA 1.5. Due to rebuttal format constraints, we are unable to include full results and plots here, but we will include them in the camera-ready version. For example, for training with image/description variants scenario with 50 epochs, we observe the following source → target accuracy:

We observe a similar asymmetric cross-modal transfer pattern. However, LLaVA exhibits slightly lower transferability in both directions compared to Gemma-3. In the future version, we plan to include more model families.

More broadly, one of our goals is to demonstrate that a cross-modal memorization gap emerges even with state-of-the-art open-source VLMs like Gemma-3. We hope our pipeline serves as a useful evaluation routine for assessing newly developed multimodal models.

Minor suggestions

Thank you for your helpful suggestions to improve the clarity of our paper. We will incorporate them in the camera-ready version.

We hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions.

We sincerely appreciate your valuable feedback and the time you've dedicated to providing it. Below, we address specific points you raised:

Reliance on fully synthetic personas may limit external validity, as real-world data distributions are not evaluated.

We agree that using real images would be ideal. However, we use fully synthetic data to prevent data contamination and ensure annotation accuracy. With real images, it is often unclear whether the associated knowledge has already appeared during pretraining or in other data points, making it difficult to isolate cross-modal memorization effects. Our synthetic setup enables a controlled investigation of memorization without the confounding risk of training data leakage. Additionally, using real human faces might raise serious privacy and consent concerns.

To minimize bias and ensure high data quality, we intentionally selected the state-of-the-art generative models in our paper. We used Imagen 3 for image generation for its top-tier performance in photorealism and content diversity (https://lmarena.ai/leaderboard/text-to-image/pre-generated-prompts-surprise-me-only). In contrast, alternatives like Midjourney or GPT-4o’s native generation often exhibit stylistic biases. For textual descriptions, we used Gemini 2.0, conditioning on structured persona attributes to significantly reduce hallucinations and maintain consistency across modalities.

The study focuses solely on name-recall tasks.

We focused on the name-recall task because it serves as a straightforward proxy for factual memorization that is easy to verify, allowing us to minimize confounding variables during data generation and training. Additionally, we included results in the paper on recalling a “favorite number” as a secondary fact to demonstrate generalization beyond names.

We agree that extending the study to include more abstract or higher-level knowledge—such as events or relationships—would provide valuable insights. Designing such experiments requires a different pipeline for data construction, and we view this as a promising direction for future work.

All experiments use a single VLM family (Gemma-3-IT), so conclusions may not generalize to other architectures or pretraining regimes.

We have also run additional experiments with LLaVA 1.5. Due to rebuttal format constraints, we are unable to include full results and plots here, but we will include them in the camera-ready version. For example, for training with image/description variants scenario with 50 epochs, we observe the following source → target accuracy:

We observe a similar asymmetric cross-modal transfer pattern. However, LLaVA exhibits slightly lower transferability in both directions compared to Gemma-3. In the future version, we plan to include more model families.

More broadly, one of our goals is to demonstrate that a cross-modal memorization gap emerges even with state-of-the-art open-source VLMs like Gemma-3. We hope our pipeline serves as a useful evaluation routine for assessing newly developed multimodal models.

The limitations of this study is not discussed in the paper.

We appreciate the reviewer pointing this out. While the limitations are discussed at the end of Section 4, we agree that they could be more clearly highlighted. Specifically, we acknowledged two main limitations: (1) Pur study focuses solely on image-text cross-modal knowledge transfer, whereas other modalities (e.g., audio, video) remain unexplored; and (2) We only evaluated baseline mitigation strategies, leaving more advanced architectural or algorithmic approaches to future work.

To improve clarity, we will add a dedicated "Limitations" subsection title in the camera-ready version to make these points more explicit and accessible.

We hope our response can resolve your concerns regarding our paper. Please let us know if you have any more questions.

Thanks for authors' detailed and timely rebuttal. All of my questions have been answered with good answers and extensive qualitative results, and I am happy to raise my score.