GMAIL: Generative Modality Alignment for generated Image Learning

We sincerely appreciate the reviewer's feedback and the questions raised. Below, we address and resolve each of these questions in our responses.

Domain Adaptation Literature

Our approach is conceptually related to domain adaptation techniques [a,b], where the goal is to align feature spaces between source and target domains. However, GMAIL focuses on a unique modality gap between synthetic and real images, both from the same caption domain, requiring a different solution strategy. While traditional DA aligns features across different datasets, our method aligns representations across modalities (real vs. generated) within the same semantic context, using caption anchors.

[a] Tzeng et al., Adversarial Discriminative Domain Adaptation, CVPR 2017

[b] Ganin & Lempitsky, Unsupervised Domain Adaptation by Backpropagation, ICML 2015

References to Data Synthesis for Vision Tasks

Thank you for the suggestion! We’ve now added and discussed all three works in Related Work:

• DatasetGAN (CVPR’21) [1]: Efficient labeled data generation via GANs and semantic segmentation.
• SegGen (ECCV’24) [2]: Mask2Img diffusion pipelines for boosting segmentation.
• DiffuMask (ICCV’23) [3]: Pixel-level annotations via text-to-mask-to-image synthesis.

We emphasize that GMAIL is complementary to these efforts:

• They focus on generating high-quality labeled datasets.
• We focus on bridging modality gaps when using synthetic data for training vision-language models.

[1] DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort. CVPR 2021

[2] SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesis. ECCV 2024

[3] DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models. ICCV 2023

Generator Generality

We have now conducted experiments using FLUX, which introduces a more powerful and differently parameterized generation pipeline compared to SD2. These additional results allow us to test GMAIL's robustness to shifts in generator-specific artifacts. The performance improvements remain consistent with FLUX, indicating robust alignment across varying artifact styles and photorealism levels.

Robustness to Adversarial or Out-of-Distribution (OOD) Inputs

Thank you for the insightful observation. To evaluate the robustness of our alignment-based GMAIL framework, we conducted two evaluations.

1. We evaluate GMAIL and baselines on ImageNet-A and ImageNet-R — well-established OOD benchmarks that test model resilience under distribution shifts. Naively training with generated images slightly reduces robustness, but GMAIL improves OOD generalization by aligning synthetic features more closely with real ones.

2. We tested models using AutoAttack on zero-shot classification over ImageNet-1k (100-class subset). GMAIL improves clean and adversarial robustness, suggesting that alignment avoids overfitting to synthetic artifacts, a key cause of fragility in other generative training schemes.

We sincerely appreciate the reviewer's feedback and the questions raised. Below, we address and resolve each of these questions in our responses.

Clarification

Thank you for catching this ambiguity. We've clarified both the notation and purpose in the Table below.

• Real only: Standard CLIP fine-tuned solely on real images (baseline).
• Mix: Real + generated data combined, no alignment applied.
• GMAIL: Separate encoders, aligned via our method.

Why does “real-only” improve? Because fine-tuning ClipCap on real data improves over frozen pre-trained CLIP. However, it underperforms compared to models using generated data, especially GMAIL, which preserves real performance while benefiting from synthetic expansion.

Comparison to Tasks2Sim

Great point. We have added experiments comparing Tasks2Sim in the Table below. GMAIL outperforms prior mix-only methods due to explicit modality disentanglement and alignment, rather than blending synthetic and real data blindly.

Embedding Comparisons

Thank you for the suggestion! We calculated the average Cosine Similarity Between Modalities including Real, Gen, and Text in the Table below. Real vs. Gen gap is smaller than image-text, but still substantial, justifying separate modality treatment. Although the cosine similarity scores for Real vs. Text (0.44) and Gen vs. Text (0.42) appear numerically close, both are substantially higher than the similarity between Real vs. Gen (0.25). This discrepancy shows a significant gap between real and generated images, indicating that each type of image is more semantically aligned with textual descriptions than they are with each other.

Citation of LeCun (2022)

Thank you for pointing this out. We originally referred to the broader concept of representation collapse (often called "mode collapse") in the context of over-optimization, as discussed in LeCun's vision report starting from Section 4.3 ("Training Energy-Based Models," p. 20 onwards). Specifically, LeCun describes how certain training methods risk collapsing learned representations, thereby reducing their generalization capacity.

Typo

We fixed it.

We sincerely appreciate the reviewer's feedback and the questions raised. Below, we address and resolve each of these questions in our responses.

Training Steps

Thank you for this observation. We have conducted additional experiments with equal training steps across all compared methods and included the results in the Table below. Even with identical training steps, GMAIL consistently outperforms both baselines, confirming that gains stem from our alignment design, not from additional training.

Experiments on MMMU

Thank you! We included results on MMMU, which is a benchmark that requires multimodal reasoning and visual-language alignment. GMAIL improves visual grounding and alignment with LLMs, demonstrating generalization to reasoning-based VLM tasks.

Method Description

We apologize for the confusion. We’ve made the following changes to improve clarity: Figure 1 is now redesigned and clarified with modality-specific flows and alignment stages. We added formal notation in Section 3.1 (Preliminaries) and Section 3.2 (Gen-CLIP Flow). We also introduced the key equation for alignment and clarified dual-encoder roles.

Algorithm 1

We’ve rewritten Algorithm 1 in Appendix B with clearer, formal steps and precise mathematical notation. We replaced vague language like “aligned representation​” with formal notation, and split training into two distinct phases: Gen-CLIP Flow with LoRA adaptation and Inference/Transfer to Vision-Language Models (CLIPCap, LLaVA, etc.)

Sentence in Line 188-190

We use a dual projection structure. During inference, real images are encoded via the original encoder $f_r$ (not $f_g$). This avoids degradation from overfitting to synthetic artifacts in $f_g$. The aligned model still benefits from synthetic training via shared projection space.

References to Data Synthesis for Vision Tasks

Thank you for the suggestion! We’ve now added and discussed all three works in Related Work:

• DatasetGAN (CVPR’21): Efficient labeled data generation via GANs and semantic segmentation.

• SegGen (ECCV’24): Mask2Img diffusion pipelines for boosting segmentation.

• DiffuMask (ICCV’23): Pixel-level annotations via text-to-mask-to-image synthesis.

We emphasize that GMAIL is complementary to these efforts:

• They focus on generating high-quality labeled datasets.
• We focus on bridging modality gaps when using synthetic data to train vision-language models.

Typos

We carefully proofread the entire paper and corrected all identified typos.

We sincerely appreciate the reviewer's feedback and the questions raised. Below, we address and resolve each of these questions in our responses.

Model Collapse

We agree. To clarify the severity and nature of the issue, we added a direct comparison between models trained on: Real images only (baseline), Generated images without alignment, Generated images with our GMAIL alignment. Training directly on generated images without alignment reduces generalization to real-world test data. GMAIL mitigates this through modality-aware learning.

Without alignment, performance improves on synthetic data, but fails to transfer well to real data. GMAIL avoids overfitting to synthetic artifacts by learning aligned representations.

More Analysis

Thank you for the suggestion. We now refer the term “model collapse” to better reflect what we observe: a divergence from real-image generalization, rather than complete failure. We also show latent space shifts. Without GMAIL, embeddings for the same image-caption pair differ significantly across real and synthetic inputs.

Embedding Space Collapse Without Alignment To complement the cosine similarity, we include a t-SNE visualization (Figure 3, Appendix), which shows that real vs. generated images cluster apart without GMAIL, but overlap with GMAIL, supporting the hypothesis of latent space misalignment and collapse risk.

Scalability and Training Stability with GMAIL GMAIL introduces modest memory overhead but significantly improves convergence and stability, especially in large-scale training. We empirically found that GMAIL improves convergence speed and representation robustness with fewer training steps. | Method | Synthetic Data | Steps ↓| |:----------------:|:---------------:|:---------------:| | CLIP (gen only) | ✔ | 70k | | GMAIL (ours) | ✔ | 50k |

Effectiveness on Downstream Tasks We included experiments on: Alignment improvements to SigLIP models (Table 9) and Visual QA on ScienceQA (Table 10) in the appendix. We also include the results on MMMU benchmark, which is a benchmark that requires multimodal reasoning and visual-language alignment, in the Table below. GMAIL improves visual grounding and alignment with LLMs, demonstrating generalization to reasoning-based VLM tasks.

Naming

This acronym was chosen for memorability, but we understand the concern. We’ve added a footnote clarifying no relation to Google.

Font and Typos

We fixed the typo in Line 84, and Figure 1 now uses CMU Serif, is fully vectorized (PDF), and screen-reader accessible.