CycleAug: Cycle-Consistent Visual Augmentation for Large Multimodal Models

W5: The authors should provide discussions about the failure cases and their limitations.

We appreciate the opportunity to discuss the limitations.

(1) Text-rich VQA: The current version might not handle well with text-rich VQA. This stems from the common limitations of the Stable Diffusion models in generating characters, which hinders its performance on text-focused QA tasks like TextVQA. However, we believe this limitation is not inherent to our pipeline but rather tied to the current capabilities of image generation models. As these models continue to improve in text rendering (such as ideogram and Text-Diffuser2), we expect this issue to be resolved.

(2) Misaligned Samples: This occurs when the cycle-consistency sampling process does not completely filter out all misaligned images. While our consistency judge effectively identifies and removes most misaligned samples with respect to the question-answer pair, occasional errors arise from imperfections in the underlying vision-language model and image generation model. Nonetheless, our carefully designed consistency judge successfully identifies the majority of valid samples in an automatic and efficent way, improving the model's performance on VQA tasks.

Through analysis, we observe that the negative impact of erroneous images is most pronounced in datasets requiring precise fine-grained visual-text alignment, such as text-rich scenarios. Conversely, the impact is minimal on benchmarks with a broader focus, such as GQA, MMB, VQAv2, where diversity in visual content is a more critical factor.

W4: At least 3-4 MLLM models should be performed to evaluate the proposed method fairly. authors should provide a detailed complexity and efficiency analysis of the cycle-consistency visual augmentation process.

To address the reviewer’s concerns, we have conducted additional experiments across multiple MLLMs, including LLaVA-LLaMA3, LLaVA-v1.5-13B, and MGM-2B:

Performance Gains: Our method consistently outperforms baselines, with notable improvements on GQA, MMB, and MM-Vet benchmarks. Average performance gains are +1.19%, +0.34%, and +0.23% for MGM-2B, LLaVA-13B, and LLaVA-LLaMA3, respectively.

We acknowledge the importance of providing a detailed complexity and efficiency analysis for the cycle-consistency visual augmentation process. This analysis demonstrates that our method is efficient relative to the scale of augmentation and its benefits:

Cost Analysis:

The computational cost is estimated based on the FLOPs and latency of each component, averaged across 200 random samples, and scaled to the total dataset size. The measurements are as follows:

[1] Visual autoregressive modeling: Scalable image generation via next-scale prediction. 2024.

FLOPs and time-cost distribution: We measure the FLOPs and time cost of CycleAug to evaluate its computational efficiency. As shown in Table, the image generation step (Stable Diffusion 3) accounts for the highest computational cost, requiring 133.9 GPU-hours due to its high FLOPs (87.9T) and latency (4.1s/sample). Image captioning and VQA (LLaVA-Llama-3) strike a balance, 91.0 GPU-hours, with moderate FLOPs (8.3T) and latency (1.2s/sample). The consistency judgment step (Sbert-Model) is the most efficient, contributing only 3.7 GPU-hours.This analysis highlights that image generation is the main bottleneck, while captioning and consistency judgment remain computationally efficient. Optimizing the generation step could significantly reduce overall cost, improving scalability for larger datasets.

Large Room for FLOPs reduction: Although the FLOPs and time costs are significant for Stable Diffusion 3, we also measured the performance of VAR, an autoregressive-based text-to-image generation method with comparable quality to diffusion-based methods (FID: 1.8 vs. 1.56). VAR requires only 22.9 GPU hours to generate 117k images, making it both efficient and scalable to larger datasets.

Since our current focus is on demonstrating the proof of concept, we have not emphasized efficiency. However, the performance of VAR highlights the potential for our pipeline to achieve high efficiency, especially when integrated with more computationally efficient models.

Conclusion: CycleAug's computational cost is significant, primarily due to the image generation step, but the filtering process ensures high-quality data critical for performance improvements. Efficient alternatives like VAR, with comparable quality and lower costs, offer potential for optimization, making CycleAug more scalable for larger datasets.

We appreciate the reviewer's constructive suggestions and apologize for the delayed response, as we have been preparing experiments to thoroughly address your concerns. In addition, we have revised our manuscript based on your suggestions.

W1: The novelty is weak. The consistency sampling is directly inspired by the existing cross-modal consistency alignment framework. Although the idea is straightforward, there is no special or novel technical design, weakening the novelty of the proposed method.

We thank the reviewer for the opportunity to clarify the novelty and contributions of our work. However, we respectfully disagree with the assertion that our method is directly inspired by existing cross-modal consistency alignment frameworks.

We would greatly appreciate it if the reviewer could elaborate further on these frameworks for better context. Our work does not aim to develop a new multimodal alignment approach. To clarify, our work represents the first attempt to augment the training of large multimodal models using cycle-consistency. To the best of our knowledge, no prior research has explored this approach for training large multimodal models.

W2: Utilizing technique-irrelevant formulations and proof is not necessary.

Thank you for highlighting the distinction between our mathematical formulation and the implementation techniques. While we acknowledge that the mathematical formulation may not directly align with every aspect of the implementation, it serves as a precise and clear expression of our underlying motivation.

Inspired by analogical reasoning, our method aims to encourage feature-space consistency for semantically similar yet visually distinct image pairs. This concept is captured in Lemma 2, providing a theoretical view for our approach.

To ensure that the paper remains focused and accessible, we have included the full proof in the appendix while presenting the key conclusions in the main text. This strategy maintains the clarity and flow of the article while offering a deeper theoretical perspective for interested readers. Furthermore, the experimental results validate the practical effectiveness of this regularization, demonstrating its importance in enhancing the performance of the model.

W3: generating synthetic texts via the loop of image-text-image to provide new aspects of the same image

Thank you for the thoughtful suggestion regarding generating synthetic texts via the loop of image-text-image to provide new perspectives on the same image. While our current work focuses on visual augmentation through the text-image-text loop, we agree that exploring the reverse loop (image-text-image) is a promising direction for generating diverse textual data. This idea could enrich multimodal learning by introducing additional dimensions of understanding.

However, we respectfully note that this direction falls beyond the scope of our current work, which is specifically centered on visual augmentation and its impact on multimodal learning models. We kindly request that aspects unrelated to the core contributions of the paper not be considered as weaknesses, as they do not detract from the novelty or rigor of our proposed method. We look forward to exploring this idea further in future research.

Dear Reviewer,

If our response does not address your remaining concerns, please let us know, and we will address them promptly before the rebuttal period concludes. In addition, we have revised our manuscript based on your suggestions.

Thank you!

W2 & Q2: Since the authors use better MLLM in the CYCLEAUG framework, is the performance gain from the CYCLEAUG framework itself, or from distilling knowledge from the better MLLM (LLaVA-NeXT) to the MLLM we are training (LLaVA-1.5-7B)

Thank you for pointing out this important consideration. We acknowledge the concern about the difference in models used for captioning and consistency judgment compared to the target model being trained.

In our current experimental setup, we use stronger models (e.g., LLaVA-NeXT) for image captioning and consistency judgment to ensure high-quality augmented data. However, we want to emphasize that our approach is not limited to this setting. It can also operate effectively in a self-training scenario where the captioner and judge are the same as the target model, eliminating the need for a better external model.

To address this, we conducted additional experiments using LLaVA-v1.5-7B as both the captioner and the consistency judge. These experiments demonstrate that even under a self-training setting, our method provides consistent improvements on the VQA task.

The results from the self-training experiment demonstrate that CycleAug provides consistent improvements even when using the same model (LLaVA-v1.5-7B) as both the captioner and consistency judge. This eliminates the need for a stronger external model, showing that the performance gains stem from the CycleAug framework itself rather than distilling knowledge from a better MLLM.

We appreciate the reviewer’s constructive suggestions and apologize for the delayed response, as we have been preparing experiments to thoroughly address your concerns. In addition, we have revised our manuscript based on your suggestions.

W1 & Q1: approximate computational cost for the data augmentation by CYCLEAUG in the main experiment

We appreciate the reviewer’s suggestion to report the computational cost and filtering proportions, which we have now calculated. Below, we detail the FLOPs, latency, and overall time for each stage of our process, along with the proportion of images filtered out by cycle-consistency sampling:

Cost Analysis: The computational cost is estimated based on the FLOPs and latency of each component, averaged across 200 random samples, and scaled to the total dataset size. The measurements are as follows:

[1] Visual autoregressive modeling:Scalable image generation via next-scale prediction. 2024.

Filtering Proportion: Out of the 273,144 images initially generated, 117,576 images (43%) were retained after applying cycle-consistency sampling. This step effectively eliminates 57% of the images, ensuring the final dataset is both diverse and of high quality.

FLOPs and time-cost distribution: We measure the FLOPs and time cost of CycleAug to evaluate its computational efficiency. As shown in Table, the image generation step (Stable Diffusion 3) accounts for the highest computational cost, requiring 133.9 GPU-hours due to its high FLOPs (87.9T) and latency (4.1s/sample). Image captioning and VQA (LLaVA-Llama-3) strike a balance, processing 273,144 samples in 91.0 GPU-hours, with moderate FLOPs (8.3T) and latency (1.2s/sample). The consistency judgment step (Sbert-Model) is the most efficient, contributing only 3.7 GPU-hours.This analysis highlights that image generation is the main bottleneck, while captioning and consistency judgment remain computationally efficient. Optimizing the generation step could significantly reduce overall cost, improving scalability for larger datasets.

Large Room for FLOPs reduction: Although the FLOPs and time costs are significant for Stable Diffusion 3, we also measured the performance of VAR, an autoregressive-based text-to-image generation method with comparable quality to diffusion-based methods (FID: 1.8 vs. 1.56). VAR requires only 22.9 GPU hours to generate 117k images, making it both efficient and scalable to larger datasets.

Since our current focus is on demonstrating the proof of concept, we have not emphasized efficiency. However, the performance of VAR highlights the potential for our pipeline to achieve high efficiency, especially when integrated with more computationally efficient models.

Conclusion: CycleAug's computational cost is significant, primarily due to the image generation step, but the filtering process ensures high-quality data critical for performance improvements. Efficient alternatives like VAR, with comparable quality and lower costs, offer potential for optimization, making CycleAug more scalable for larger datasets.

Dear Reviewer,

If our response does not address your remaining concerns, please let us know, and we will address them promptly before the rebuttal period concludes. In addition, we have revised our manuscript based on your suggestions.

Thank you!

Q1: integrating the question-answer pair into the prompt may improve the proportion of consistent triplets in all generated images

Thank you for the valuable suggestion regarding integrating QA pairs into the image captioning process. This approach is promising but introduces two significant challenges:

1. Higher Requirements on Multi-Modal LLMs: Generating coherent captions that effectively incorporate both QA pairs and images demands more advanced reasoning capabilities from multi-modal LLMs.
2. Increased Computational Costs: The inclusion of longer QA pairs adds more tokens to process, significantly increasing the computational overhead.

To assess the effectiveness of this method, we conducted an experiment comparing image-only and QA-integrated captioning under the same amount of synthetic images. Results are as follows:

Performance: QA-integrated captioning shows marginal improvements in some benchmarks (e.g., MMMU, GQA, and SEED-img) but slightly lower results in others (e.g., MME). The overall average remains comparable (62.18 vs. 62.33).

Computational Cost: The time required for QA-integrated captioning is significantly higher, increasing from 66 GPU-hours to 254 GPU-hours, reflecting the additional processing of QA pairs.

Conclusion: While QA-integrated captioning may slightly enhance alignment between captions and QA pairs, the marginal performance gains do not justify the steep increase in computational costs under the current framework. Future advancements in computationally efficient multi-modal models may make this approach more viable.

Q2: additional experiments and analysis on how various MLLM and diffusion models will affect the final performance on the benchmark

Thank you for the constructive suggestion. As highlighted in our response to W1, our pipeline demonstrates adaptability to various architectures, parameter scales, and visual encoders. To further extend our analysis, we are currently experimenting with autoregressive-based model, VAR[1]. Such experiment will help clarify how diverse generative models impact the effectiveness and performance of our framework across benchmarks. We will provide detailed results and analysis in the updated manuscript.

[1] Visual autoregressive modeling:Scalable image generation via next-scale prediction. 2024.

We appreciate the reviewer’s constructive suggestions and apologize for the delayed response, as we have been preparing experiments to thoroughly address your concerns. In addition, we have revised our manuscript based on your suggestions.

W1:Performance should be evaluated using more diverse backbones, which can better assess the robustness and effectiveness of the proposed framework

To address the reviewer’s concerns, we have conducted additional experiments across multiple MLLMs, including LLaVA-LLaMA3, LLaVA-v1.5-13B, and MGM-2B:

CycleAug brings gains on more backbones: Our method consistently outperforms baselines, with notable improvements on GQA, MMB, and MM-Vet benchmarks. Average performance gains are +1.19%, +0.37%, and +0.23% for MGM-2B, LLaVA-13B, and LLaVA-LLaMA3, respectively.

W2: This paper should clarify the relevance to the existing work [1]. [1] (CVPR 2019) Cycle-Consistency for Robust Visual Question Answering

We appreciate the opportunity for further clarification. While both approaches use the term "cycle-consistency," our method differs significantly in focus and implementation:

Focus on Visual Augmentation: Our method leverages visual foundation models, such as Stable Diffusion 3, to augment visual data, enhancing the diversity and quality of training datasets. In contrast, the work in [1] emphasizes augmenting Q&A pairs by designing a specific loss function (Eq. 1 in [1]) to explicitly enforce consistency between generated and original Q&A pairs.

Application to Large Multimodal Models: Our approach is tailored to improve the performance of large multimodal models, comprising a vision encoder, multimodal projector, and large language model, trained using token prediction loss. In contrast, [1] focuses on validating their method across various models, which do not include large-scale multimodal architectures.

Dear Reviewer,

If our response does not address your remaining concerns, please let us know, and we will address them promptly before the rebuttal period concludes. In addition, we have revised our manuscript based on your suggestions.

Thank you!

W2: lacks an in-depth exploration of alternative dialogue generation strategies,which could restrict the analysis of image diversity and relevance, impacting the overall model performance.

We appreciate the reviewer’s comment highlighting the potential impact of exploring alternative dialogue generation strategies. Our method focuses on generating diverse images by captioning real images using a VLM-captioner. Since it is not feasible for the VLM-captioner to capture every detail of an image, the Stable Diffusion 3 (SD3) model synthesizes high-fidelity images based on its own distribution. This process inherently introduces diversity, effectively serving as data augmentation.

To ensure alignment between the synthesized images and the QA pairs, our method employs a consistency judge. This guarantees both diversity and relevance, resulting in an effective data augmentation strategy for improving model performance.

While dialogue generation strategies are not our focus, we agree that such strategies have the potential to further enhance data augmentation.

W3:In addition to the direct sampling method, there are numerous data augmentation techniques applicable to the field of multimodal learning.

Thank you for highlighting the importance of comparative experiments with other augmentation techniques. We agree that incorporating such comparisons is valuable for assessing CycleAug's performance and showcasing its potential advantages.

We conducted experiments with transformation that composes of commonly used augmentation techniques including random cropping and resizing, rotation, and color jittering. The results are summarized below:

Such composed augmentation diversify the visual appearance of images but negatively impact concept alignment with QA pairs. For instance: Rotation can alter the spatial relationships in images, which are crucial for tasks requiring positional reasoning. CycleAug, in contrast, our method achieves higher alignment between images and QA pairs by leveraging cycle-consistency sampling and maintaining the semantic coherence of the data.

Conclusion: The comparative experiments demonstrate that CycleAug outperforms traditional augmentation techniques by preserving concept alignment while diversifying the data. These results highlight the importance of maintaining semantic consistency in multimodal learning tasks.

We appreciate the reviewer’s constructive suggestions and apologize for the delayed response, as we have been preparing experiments to thoroughly address your concerns. In addition, we have revised our manuscript based on your suggestions.

W1: a detailed discussion on the specific computational costs and resource consumption of the proposed method

We appreciate the reviewer’s suggestion to report the computational cost and filtering proportions, which we have now calculated. Below, we detail the FLOPs, latency, and overall time for each stage of our process, along with the proportion of images filtered out by cycle-consistency sampling:

Cost Analysis: The computational cost is estimated based on the FLOPs and latency of each component, averaged across 200 random samples, and scaled to the total dataset size. The measurements are as follows:

[1] Visual autoregressive modeling:Scalable image generation via next-scale prediction. 2024.

Filtering Proportion: Out of the 273,144 images initially generated, 117,576 images (43%) were retained after applying cycle-consistency sampling. This step effectively eliminates 57% of the images, ensuring the final dataset is both diverse and of high quality.

FLOPs and time-cost distribution: We measure the FLOPs and time cost of CycleAug to evaluate its computational efficiency. As shown in Table, the image generation step (Stable Diffusion 3) accounts for the highest computational cost, requiring 133.9 GPU-hours due to its high FLOPs (87.9T) and latency (4.1s/sample). Image captioning and VQA (LLaVA-Llama-3) strike a balance, processing 273,144 samples in 91.0 GPU-hours, with moderate FLOPs (8.3T) and latency (1.2s/sample). The consistency judgment step (Sbert-Model) is the most efficient, contributing only 3.7 GPU-hours.This analysis highlights that image generation is the main bottleneck, while captioning and consistency judgment remain computationally efficient. Optimizing the generation step could significantly reduce overall cost, improving scalability for larger datasets.

Large Room for FLOPs reduction: Although the FLOPs and time costs are significant for Stable Diffusion 3, we also measured the performance of VAR, an autoregressive-based text-to-image generation method with comparable quality to diffusion-based methods (FID: 1.8 vs. 1.56). VAR requires only 22.9 GPU hours to generate 117k images, making it both efficient and scalable to larger datasets.

Since our current focus is on demonstrating the proof of concept, we have not emphasized efficiency. However, the performance of VAR highlights the potential for our pipeline to achieve high efficiency, especially when integrated with more computationally efficient models.

Conclusion: CycleAug's computational cost is significant, primarily due to the image generation step, but the filtering process ensures high-quality data critical for performance improvements. Efficient alternatives like VAR, with comparable quality and lower costs, offer potential for optimization, making CycleAug more scalable for larger datasets.

Dear Reviewer,

If our response does not address your remaining concerns, please let us know, and we will address them promptly before the rebuttal period concludes. In addition, we have revised our manuscript based on your suggestions.

Thank you!

W4: The paper would benefit from reporting the cost (e.g., FLOPS, latency) required to produce these images, as well as the proportion of images filtered out by cycle consistency sampling. We appreciate the reviewer’s suggestion to report the computational cost and filtering proportions, which we have now calculated. Below, we detail the FLOPs, latency, and overall time for each stage of our process, along with the proportion of images filtered out by cycle-consistency sampling:

Cost Analysis: The computational cost is estimated based on the FLOPs and latency of each component, averaged across 200 random samples, and scaled to the total dataset size. The measurements are as follows:

[1] Visual autoregressive modeling:Scalable image generation via next-scale prediction. 2024.

Filtering Proportion: Out of the 273,144 images initially generated, 117,576 images (43%) were retained after applying cycle-consistency sampling. This step effectively eliminates 57% of the images, ensuring the final dataset is both diverse and of high quality.

FLOPs and time-cost distribution: We measure the FLOPs and time cost of CycleAug to evaluate its computational efficiency. As shown in Table, the image generation step (Stable Diffusion 3) accounts for the highest computational cost, requiring 133.9 GPU-hours due to its high FLOPs (87.9T) and latency (4.1s/sample). Image captioning and VQA (LLaVA-Llama-3) strike a balance, processing 273,144 samples in 91.0 GPU-hours, with moderate FLOPs (8.3T) and latency (1.2s/sample). The consistency judgment step (Sbert-Model) is the most efficient, contributing only 3.7 GPU-hours.This analysis highlights that image generation is the main bottleneck, while captioning and consistency judgment remain computationally efficient. Optimizing the generation step could significantly reduce overall cost, improving scalability for larger datasets.

Large Room for FLOPs reduction: Although the FLOPs and time costs are significant for Stable Diffusion 3, we also measured the performance of VAR, an autoregressive-based text-to-image generation method with comparable quality to diffusion-based methods (FID: 1.8 vs. 1.56). VAR requires only 22.9 GPU hours to generate 117k images, making it both efficient and scalable to larger datasets.

Since our current focus is on demonstrating the proof of concept, we have not emphasized efficiency. However, the performance of VAR highlights the potential for our pipeline to achieve high efficiency, especially when integrated with more computationally efficient models.

Conclusion: CycleAug's computational cost is significant, primarily due to the image generation step, but the filtering process ensures high-quality data critical for performance improvements. Efficient alternatives like VAR, with comparable quality and lower costs, offer potential for optimization, making CycleAug more scalable for larger datasets.

Q1: generating an image with SD where the starting point is a noisy anchor image

We thank for the constructive suggestion. Starting from a noisy anchor image is indeed an intriguing idea and holds potential for improving consistency with QA pairs. However, implementing this approach is nontrivial, as Stable Diffusion denoises in the latent feature space rather than the raw pixel space. This difference adds complexity to initializing the process with a noisy anchor image.

While this idea is promising, we plan to explore it in future work to address the technical challenges and assess its impact on maintaining consistency and improving generation quality. Thank you again for the insightful suggestion.

We appreciate the reviewer’s constructive suggestions and apologize for the delayed response, as we have been preparing experiments to thoroughly address your concerns. In addition, we have revised our manuscript based on your suggestions.

W1: concrete efforts to resolve homogeneous content and the can not handle negation word:

We appreciate the opporunity to clarify the issues of (1) homogeneous content and (2) the inability to handle negation words in image generation. The two issues primarily arise in direct sampling because the generated images are synthesized solely based on brief QA pairs. This brevity often lacks the detailed information needed to produce rich and diverse content, resulting in homogeneous outputs. While negation-related challenges could be solved by negative prompt for (2), the issue of homogeneous content (1) persists and may even be exacerbated. To address these limitations, our work adopts a “complete captioning to synthesizing” framework. This approach involves leveraging LLMs to generate detailed and context-rich captions from images. The enhanced captions mitigate the issue of homogeneous content by introducing more diverse and descriptive information. Moreover, this framework inherently resolves the negation problem, as the captioning process guided by LLMs naturally converts complex or negated expressions into straightforward and explicit descriptions.

In summary, we are fully aware of the concerns raised and we emphasize that our proposed cycle-consistency sampling framework specifically targets and resolves these issues by enhancing image diversity and addressing negation through a robust captioning pipeline.

W 2&3: The performance gain is quite limited; for example, in Table 3, the augmented images even have a negative impact on the VQAT dataset.

We appreciate the reviewer’s observation on text-rich VQA tasks. This stems from the common limitations of the Stable Diffusion models in generating characters, which hinders its performance on text-focused QA tasks like TextVQA. However, we would like to highlight that the majority of our visual augmentation is applied to COCO and GQA datasets, which focus on broader visual reasoning tasks. The efficacy of our augmentation is supported by improvements in accuracy on related benchmarks, including GQA (+1.9%), MMB (+2.7%),MMB-CN(+2.9%) and MMB-CN (+0.5%). These results demonstrate that our method effectively enhances performance on major distributions, even though its capability for text-related QA tasks remains limited due to the text-rendering constraints of Stable Diffusion 3. We also note that the limitations with text-rich VQA tasks have the potential to be addressed with the advance of image generation models (such as ideogram and Text-Diffuser2), paving the way for better support of text-rich scenarios in the future.

Dear Reviewer,

If our response does not address your remaining concerns, please let us know, and we will address them promptly before the rebuttal period concludes. In addition, we have revised our manuscript based on your suggestions.

Thank you!