Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation

Q1 (Weaknesses 1): Similar approaches have already been introduced in prior work FuseMix [1] and should be discussed.

A1: Thank you for mentioning this important prior work. FuseMix is a latent data augmentation approach proposed to facilitate efficient multimodal alignment. The distinction between FuseMix and our method lies in both motivation and mechanism. In the final version, we will include a dedicated section for this discussion.

• Motivation: FuseMix assumes that well‑pretrained unimodal encoders already capture rich semantics, leveraging them for multimodal alignment without expensive full‑model training. It employs latent data augmentation to mitigate data scarcity and the limited capacity of lightweight adapters. In contrast, our method targets the practical challenge of modality collapse in multimodal dataset distillation. Through theoretical analysis, we show that Representation Blending alleviates this issue, enhancing intra‑modal diversity and cross‑modal alignment.

• Mechanism: FuseMix is essentially a data augmentation method, where even modality‑agnostic perturbations such as Gaussian noise can also boost baseline performance. In contrast, our approach directly addresses modality collapse. As shown in Section 3.2, Gaussian noise introduces semantically meaningless signals that enlarge the modality gap and degrade cross‑modal alignment, leading to a performance drop. These results underscore the distinctiveness of our design for multimodal dataset distillation (MDD).

Q2 (Weaknesses 2): The authors used intra-modal similarity and modality gap but did not detail their individual impact on performance.

A2: Thank you for the question. The intra-modal similarity (Sim), concentration ratio (CR), and modality gap (Gap) are key metrics for evaluating the quality of the distilled dataset. In general, smaller Sim and CR indicate greater intra-modal diversity, while a smaller Gap reflects stronger cross-modal alignment. Together, these factors lead to better retrieval performance. However, as these metrics are interdependent, it is difficult to vary one without affecting the others. So here we provide two cases to generally assess their individual effects.

• Distilling 100 pairs on Flickr-30k using NFNet+BERT.

• Baseline vs. Case 1: A substantial reduction in intra-modal similarity (CR from 0.95 → 0.54) led to a 2.1% increase in IR@10 and a 2.4% increase in TR@10.

• Case 1 vs. Case 2: A further notable reduction in the cross-modal gap (0.318 → 0.23) yielded an additional 7.3% increase in IR@10 and 3.9% in TR@10 compared with Case 1.

These results indicate that reducing intra-modal concentration, and narrowing the modality gap, provides complementary benefits and leads to substantial performance gains.

Q3 (Weaknesses 3): Some experiments on unimodal task will strengthen the paper.

A3: Thank you for the insightful suggestion. We evaluated the proposed Representation Blending (RB) in the unimodal setting by adopting SRe2L [2] as the baseline and integrating RB into its distillation process. RB effectively reduces feature redundancy and promotes more class-discriminative representations. As shown below, RB consistently improves SRe2L’s performance across various IPC budgets.

• Distill CIFAR-100 with ResNet-18

Q4 (Questions 1): To reach the full performance, roughly how many samples are needed? How does this number vary across different methods? Hope to see the plot with more numbers.

A4: We extended the comparison to larger pair settings and provide a detailed table since figures are not allowed in the rebuttal. Both LoRS and our method show only marginal gains as pairs increase, reflecting the well-known saturation issue in dataset distillation where added pairs bring limited diversity. Nevertheless, our method maintains consistent superiority over LoRS under the same conditions. In the future, we will further explore ways to overcome this saturation toward lossless or nearly lossless MDD.

• NFNet + Bert on Fliker-30k dataset. The model trained on the full dataset performs: IR@1=23.16, IR@5=53.98, IR@10=66.62; TR@1=33.8, TR@5=65.7, TR@10=76.9.

• NFNet + Bert on COCO dataset. The model trained on the full dataset performs: IR@1=14.6, IR@5=38.9, IR@10=53.2; TR@1=20.6, TR@5=46.8, TR@10=61.3.

Q5 (Questions 2): How's the performance with stronger base models, like DinoV2, or BGE, NV-Embed?

A5: Thank you for the question. We have evaluated our approach with more powerful encoders, like DiNo-v2 (85,798,656 parameters) for vision and BGE-1.5 (109,482,240 parameters) for text. The results demonstrate that our method remains effective when scaling model capacity.

• DiNo-v2 + BGE-1.5 on COCO dataset. The model trained on the full dataset performs: IR@1=22.70, IR@5=51.13, IR@10=65.26, TR@1=31.04, TR@5=61.96, TR@10=74.1.

DiNo-v2: https://huggingface.co/timm/vit_base_patch16_224.dino

BGE-1.5: https://huggingface.co/BAAI/bge-base-en-v1.5

Q6 (Questions 3): How's the zero-shot performance, how well can the trained model generalize to other datasets?

A6: Thank you for raising this point. We further validated the proposed method with zero-shot ImageNet classification (10 randomly selected classes) and OCR-relevant retrieval on TextCaps. Compared with LoRS, our method trained on only 499 distilled pairs achieves higher accuracy and retrieval performance, thereby narrowing the gap with the full-data baseline.

• ImageNet-10 zero-shot classification

• TextCaps zero-shot retrieval

Beyond zero-shot performance, we also extend our method to the large-scale LLaVA-cc3m dataset, which contains 558k image-text pairs. We train on ~60% (334k pairs) and validate on a separate 10k-pair set. The results demonstrate that our method generalizes well to large-scale datasets.

• NFNet + Bert on LLaVA-cc3m dataset. The model trained on the full dataset performs: IR@1=9.13, IR@5=25.94, IR@10=36.34, TR@1=9.49, TR@5=26.08, TR@10=37.07.

[1] Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. NeurIPS 2023.

[2] TextCaps: a Dataset for Image Captioning with Reading Comprehension, ECCV 2020.

LLaVA-cc3m: https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md

Thank you for your time and constructive comments. We have responded to all your questions as follows and hope these address your concerns.

Q1 (Weaknesses 1): It’s unclear whether the method is still effective when scaling the model and data, because all the presented experiments were conducted on small models and datasets. The IR numbers are too small compared to SOTA models (we don’t need to achieve SOTA as well, but better to be more comparable).

A1: Thank you for raising this important point. To address the concern, we have extended our method to a larger-scale setting using the LLaVA-cc3m dataset, which serves as the pretraining dataset for LLaVA and consists of 558k image-text pairs. We use approximately 60% of the data (about 334k pairs) for training and reserve a non-overlapping set of 10k pairs for validation. In addition, we evaluate our approach with more powerful encoders, including DiNo-v2 (85,798,656 parameters) for vision and BGE-1.5 (109,482,240 parameters) for text. The following results demonstrate that our method remains effective when scaling both model capacity and training data, while also showing clear superiority over the state-of-the-art competitor. We note that a performance gap against the baseline still exists, which remains a longstanding challenge. Addressing this gap will be a key focus of future work toward achieving lossless or near-lossless distillation.

• NFNet + Bert on LLaVA-cc3m dataset. The model trained on the full dataset performs: IR@1=9.13, IR@5=25.94, IR@10=36.34, TR@1=9.49, TR@5=26.08, TR@10=37.07.

• DiNo-v2 + BGE-1.5 on COCO dataset. The model trained on the full dataset performs: IR@1=22.70, IR@5=51.13, IR@10=65.26, TR@1=31.04, TR@5=61.96, TR@10=74.1.

LLaVA-cc3m: https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md

DiNo-v2: https://huggingface.co/timm/vit_base_patch16_224.dino

BGE-1.5: https://huggingface.co/BAAI/bge-base-en-v1.5

Q2 (Weaknesses 2): Given the proposed method is to reduce the modality gap, demonstrated in image and text embeddings, more zero-shot tasks could be more convincing, such as INET classification, OCR-relevant retrieval (e.g. textcaps) etc.

A2: Thank you for raising this point. To further validate the effectiveness of the reduced modality gap in our distilled dataset, we additionally evaluate zero-shot ImageNet classification and OCR-relevant retrieval on the TextCaps dataset [1]. Specifically, we randomly select 10 classes from ImageNet-1K and report the Top-1 and Top-5 zero-shot classification accuracies. For TextCaps, we measure retrieval performance over 3,166 validation samples. Here, 'Baseline' denotes the model trained on the full COCO dataset and directly evaluated on these two tasks in a zero-shot manner. LoRS and Ours correspond to models trained with only 499 distilled COCO image-text pairs. The results, summarized below, show that under the same budget, models trained on our distilled dataset outperform LoRS, narrowing the performance gap with the full-dataset baseline.

• ImageNet-10 zero-shot classification

• TextCaps zero-shot retrieval

Q3 (Questions 1): How many parameters do the models have in the experiments?

A3: Thanks for the question. In our experiments, we have considered four image encoders, including ViT, RegNet, ResNet-50, and NFNet, and two text encoders including BERT and DistilBERT (same as baseline LoRS). To evaluate the scalability of our method, we further extended it to more powerful encoders, namely DiNo-v2 for vision and BGE-1.5 for language. The selected models encompass representative architectures with varying capacities, and their parameters are summarized below.

The superiority demonstrated across these diverse models not only validates the effectiveness of the proposed method, but also highlights its scalability and generalization capability.

[1] TextCaps: a Dataset for Image Captioning with Reading Comprehension, ECCV 2020.

Thank you for your time and insightful feedback. We have provided detailed responses to all your questions below and hope they effectively resolve your concerns.

Q1 (Weaknesses 1 & Questions 1): The analysis in Appendix B relies on implicit assumptions (e.g., negative sample embeddings being approximately orthogonal) that should be explicitly stated.

A1: Thanks for the insightful suggestion. This deviation indeed relies on the assumption that both the image and text encoders are well-pretrained, such that negative pairs are approximately orthogonal in the high-dimensional embedding space. This assumption is consistent with our practical setup, where we employ ImageNet-1K–pretrained image encoders (e.g., NFNet) and BookCorpus + English Wikipedia–pretrained text encoders (e.g., BERT). The validity of this assumption has also been supported by several prior works [1, 2]. To further substantiate it, we empirically computed the relevant values, obtaining a cross-modal negative pair mean cosine similarity of 0.0003 and an intra-modal (image) mean cosine similarity of 0.004. For clarity, this assumption is explicitly emphasized in the revised version.

Q2 (Weaknesses 1): Suggests expanding the visual analysis, for instance by adding a CR‑Iteration plot, to strengthen the argument.

A2: Thank you for the suggestion. We chose intra-modal similarity for illustration because it is more intuitive to understand compared with CR. The CR–Iteration plot shows the same trend as the current similarity–Iteration plot, since CR is positively correlated with intra-modal similarity; higher intra-modal similarity corresponds to higher CR, and in particular, CR provides a more numerically salient perspective. As figure updates are not permitted during rebuttal, we will incorporate this additional plot in the final version.

Q3 (Weaknesses 2): In Table 3, the NFNet+BERT evaluation model case is excluded. For completeness, it should be added to this table as well (although it can be inferred from other tables).

A3: Thanks for pointing this out. We have included the NFNet + BERT case in Table 3 to ensure completeness and consistency across the reported results.

Table 3. Cross-architecture generalization. The distilled data are synthesized using NFNet+BERT and evaluated across different architectures. Evaluations are conducted on Flickr-30K under the 500-pair setting. For fairness, both LoRS [3] and ours synthesize one fewer pair, e.g., 499 pairs.

Q4 (Weaknesses 3): Some more minor points. 1) Tables 1 and 2 should float to the bottom of the page for readability. 2) Table 4 can also become horizontal instead of vertical, so that it can float to top after Table 3. 3) I would recommend highlighting (bolding / different color) the difference between Equations 5 and 6, since at first glance they are too similar (the difference being the use of image encoder or image projection head). 4) In line 168, “blends” -> “blend”.

A4: Thank you for the helpful suggestions. We will (1) adjust the float positions of Tables 1–2, (2) reformat Table 4 horizontally, (3) highlight the difference between Eq. 5 and Eq. 6, and (4) fix the typo “blends” → “blend” in line 168 in the final version.

Q5 (Questions 2): Also, I would be grateful if the authors could let me know whether they have tried Representation Blending in the unimodal setting (as it seems that it might be useful there as well).

A5: Thank you for the insightful suggestion. We have evaluated the proposed Representation Blending (RB) in the unimodal setting. Specifically, we adopt SRe2L [4] as the baseline and incorporate RB into its distillation process. RB effectively mitigates feature redundancy and encourages the distillation of more class-discriminative representations. The results below show that integrating RB consistently improves the performance of SRe2L across different IPC budgets.

• Distill CIFAR-100 with ResNet-18

[1] Relaxing Contrastiveness in Multimodal Representation Learning, WACV 2023.

[2] Uncovering Meanings of Embeddings via Partial Orthogonality, NeurIPS 2023.

[3] Low-Rank Similarity Mining for Multimodal Dataset Distillation, ICML 2024.

[4] Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. NeurIPS 2023.

Thank you for your constructive comments! We give point-to-point replies to your questions in the following.

Q1 (Weaknesses 1 & Questions 1): Although the authors propose an informable algorithmatic issue, the modality collapse, it would be more clear if the authors could provide a detailed example to facilitate the reader's understanding.

A1: Thank you for the question.

• Definition of Modality Collapse In our paper, modality collapse refers to a phenomenon in multimodal dataset distillation (MDD) where the distilled embeddings within the same modality become overly concentrated (i.e., exhibit low intra-modal variance), while the alignment between cross-modal embeddings (e.g., image-text pairs) is insufficient. This is visually illustrated in Figure 1, where intra-modal representations cluster tightly, yet fail to align across modalities.
• Detrimental effect of Modality Collapse Such collapse results in degraded representational diversity and weakened multimodal correspondence (as shown in Figure 2), ultimately leading to significant performance degradation when models are trained on the distilled data, compared to those trained on the original data, as quantitatively demonstrated in Tables I and II.

Q2 (Weaknesses 2 & Questions 2): The relationship between Section 3.2 and 3.3 is not clearly stated. Are they two different ways of method implementation? or do they need be utilized simultaneously?

A2: Thank you for the question. Representation Blending (Section 3.1) provides a theoretically sound solution to mitigate modality collapse by encouraging intra-modal diversity. While this strategy can effectively reduce intra-modal concentration and slightly narrow the cross-modal gap (from 0.327 in LoRS to 0.318), its impact remains limited due to the inherently asymmetric nature of the distillation process.

To address this asymmetry and further enhance cross-modal alignment, we introduce a Symmetric Projection Matching strategy (Section 3.2), which aligns the projection-head trajectories of both modalities during distillation. This method significantly reduces the modality gap (from 0.327 in LoRS to 0.230), demonstrating its effectiveness in improving cross-modal alignment.

Combining these two components achieves a synergistic effect: Representation Blending enhances intra-modal diversity, while Symmetric Projection Matching ensures cross-modal alignment, thereby significantly improving the quality of the distilled dataset.

Q3 (Weaknesses 3 & Questions 3): Fig. 9–10 show distilled images/texts but lack quantitative or qualitative analysis (e.g., diversity metrics, attention maps). Claims about "enhanced diversity" need stronger visual proof.

A3: Thank you for the constructive suggestion.

• Quantitative results: The enhanced diversity of our distilled dataset is reflected in two aspects: (1) reduced intra-modal similarity, and (2) a narrower inter-modal gap. We quantify them using the Concentration Ratio (CR), intra-modal similarity (Sim), and modality gap (Gap) as defined in the main paper. The following tables summarize these quantitative results, showing that our method substantially improves diversity compared to LoRS.

• Qualitative Visualizations: In addition to the quantitative results, the qualitative visualizations in Figure 1 clearly demonstrate that the distilled embeddings are more dispersed within modalities and better aligned across modalities. Furthermore, the right subfigure in Figure 2 depicts a cross-modal similarity heatmap (functionally equivalent to an attention map), where each image exhibits distinct attention patterns toward the text, similarly, each text also attends differently to the images. This highlights the increased instance-wise diversity in cross-modal interactions.

As figures are not permitted during the rebuttal, we will revise the final version to better highlight and elaborate on these visualizations and their implications, and provide further qualitative illustrations to substantiate our claims.

Q4 (Weaknesses 4 & 5 & Questions 4 & 5): Since LoRS is an important baseline throughout the experiment section, there should be explicit definition of LoRS in the formal part of paper. The structure of Section 3.3 can also be improved. It may be better to use a prelimiary section to introduce LoRS, while the method section should focus on the different between LoRS and RepBlend.

A4: Thank you for the valuable suggestions. In the current version, Section 2 (Preliminaries and Related Work) introduces the Multimodal Dataset Distillation (MDD) problem, contextualized by two representative baselines: LoRS and MTT-VL. The details of LoRS are also described at the beginning of Section 3.1. To improve clarity and coherence, we have restructured these sections by explicitly formalizing LoRS in the preliminaries (see below), while leaving the method section to focus on the distinctions between LoRS and RepBlend.

• Preliminaries and Related Works.

Given a large-scale image-text dataset $\mathcal{D} = \{(\mathbf{x}\_i, \boldsymbol{\tau}\_i), \mathbf{y}\_i\}_{i=1}^{|\mathcal{D}|}$, where $\mathbf{x}\_i \in \mathbb{R}^{d{\_\text{img}}}$ and $\boldsymbol{\tau}\_i \in \mathbb{R}^{d{\_\text{text}}}$ denote the $i$-th image and its paired caption representation, and each pair is independently sampled from a natural data distribution $\mathcal{P}$.

Each $\mathbf{y}_i \in \lbrace 0,1 \rbrace^{|\mathcal{D}|}$ is a one-hot vector indicating the correspondence between $\mathbf{x}\_i$ and the caption set ${\boldsymbol{\tau}\_j}\_{j=1}^{|\mathcal{D}|}$, with the $i$-th entry activated. Similar to Dataset Distillation (DD), Multimodal Dataset Distillation (MDD) also aims to minimize the loss on the original dataset using the model trained on its distilled synthetic counterpart $\mathcal{S} = \{(\tilde{\mathbf{x}}\_i, \tilde{\boldsymbol{\tau}}\_i), \tilde{\mathbf{y}}\_i \}\_{i=1}^{|\mathcal{S}|}$:

where $|\mathcal{S}| \ll |\mathcal{D}|$, and $\mathcal{L}$ denotes the contrastive learning loss. The model $f\_{\boldsymbol{\theta}}(\cdot)$ represents a CLIP-style network parameterized by $\boldsymbol{\theta}$. Each distilled sample consists of a synthetic image-text pair $(\tilde{\mathbf{x}}\_i, \tilde{\boldsymbol{\tau}}\_i)$, where $\tilde{\mathbf{x}}\_i \in \mathbb{R}^{d\_\text{img}}$ and $\tilde{\boldsymbol{\tau}}\_i \in \mathbb{R}^{d\_\text{text}}$, accompanied by a learned soft label $\tilde{\mathbf{y}}\_i$.

LoRS is a representative MDD method built upon Equation [1], where the loss function $\mathcal{L}$ is defined as:

Here, $\mathcal{B} \subset \mathcal{S}$ denotes a sampled batch. $\hat{\mathbf{y}}\_{ij}$ represents the cosine similarity between the normalized image and text embeddings, where: $\tilde{\mathbf{x}}\_i^\prime = \operatorname{Normalize}(f^{\text{imgE}}(\tilde{\mathbf{x}}\_i))$ and $\tilde{\boldsymbol{\tau}}\_j^\prime = \operatorname{Normalize}(f^{\text{textP}}(\tilde{\boldsymbol{\tau}}\_j))$ with $f^{\text{imgE}}(\cdot)$ and $f^{\text{textP}}(\cdot)$ denoting the image encoder and text projection head, respectively. Note that In LoRS [1], no image projection head is used. The threshold $\beta$ is used to determine positive and negative pairs, $\sigma(\cdot)$ denotes the sigmoid function, and $\gamma$ is the temperature. $\ell(\cdot, \cdot)$ refers to the binary cross-entropy loss. While this supervision primarily aims to mine cross-modal relationships, it inadvertently reinforces intra-modal similarities, ultimately leading to Modality Collapse.

[1] Low-Rank Similarity Mining for Multimodal Dataset Distillation, ICML 2024.