CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning

In the ablation experiments, it was observed that the performance improvement brought by CtrlSynth-img alone is minimal. Would it be possible to completely remove the generation of synthetic images and focus resources on improving the diversity and quality of synthetic text? Would this lead to consistent performance improvements across all tasks?

Response 1: Thank you for raising this point. The inclusion of image synthesis in our approach is to enable a broader range of diverse text-image synthesis paths and to complete the closed-loop pipeline. While CtrlSynth-img alone provides only minimal improvements in CLIP performance, it has significant potential to improve results in cases where the text is noisy or when the quality of synthetic text degrades. Given the modular nature of CtrlSynth, we can easily remove or replace either component depending on the use case. For CLIP specifically, we could remove CtrlSynth-img and focus on improving the synthetic text. However, for other tasks, such as long-tail vision datasets, we still need synthetic images.

The paper mentions that CtrlSynth uses a self-filtering mechanism to improve the quality of synthetic data, but it lacks detailed explanations about the implementation, such as how the alignment threshold for visual tags is selected.

Response 2: In Section 4.4 and Figure 6(a), we describe the methodology and results related to different thresholds for self-filtering. The threshold is selected based on the zero-shot accuracy of the trained CLIP models, evaluated on the ImageNet validation set. To provide further clarity, we have added additional explanations for the self-filtering process in Appendix A.5.

The paper does not explain in depth how CtrlSynth fundamentally differs from other caption augmentation methods like VeCLIP and LaCLIP. It is necessary to provide a clearer comparison, clarifying whether the increased diversity brought by the decomposition of visual tags and user control strategies is more important, or whether it is the generation of more fine-grained semantic captions that matters.

Response 3: Compared to VeCLIP and LaCLIP, the key difference is that our CtrlSynth system has more fine-grained visual tags and more diverse synthesis paths. In the ablation study (Section 4.5, Table 8), we provide evidence that the improvements of CtrlSynth come from both the use of more fine-grained semantic captions and the diverse text-image synthesis path. We also present the table below:

The experiments may be limited to a few selected models (e.g., Mistral-Nemo and Qwen2-7B). Would using larger LLMs lead to better results?

Response 4: Mistral-Nemo, with 12 billion parameters, is a larger model compared to Qwen2-7B, which has 7 billion parameters. As shown in Table 8, we observe improvements when using the larger Mistral-Nemo model. We leave the exploration of scaling to even larger models for future work.

A drawback of this method is that the data generation pipeline involves multiple different models and is not end-to-end in training, requiring substantial resources and time for building the synthetic data in the early stages.

Response 5: We believe that the modularity of our pipeline, which supports different models, is a strength rather than a drawback. It does not rely on model-specific configurations, making it flexible and adaptable. Additionally, the pipeline is currently training-free, meaning it does not require end-to-end training. Our focus is on leveraging the capabilities of pretrained models in a plug-and-play manner to facilitate data generation, rather than fine-tuning or adapting the models for specific use cases.

Practical concerns: By using several models, such as the vision tagging model, LLM, and diffusion model, the proposed method might not be efficient for scaling up to larger datasets, particularly considering the time cost associated with image synthesis. The assumption behind CtrlSynth is based on a fixed number of data samples, where the method helps a model achieve better performance than training on the original dataset. However, given the recent trends in LLM and multimodal LLM research, where pretraining data continues to scale up, the proposed method may not be scalable for very large datasets. While this is a challenge, under the current setting in the paper, CtrlSynth is indeed effective.

Response 1: Thank you for acknowledging the effectiveness of our method in the current setting. Our primary goal is to demonstrate the effectiveness of CtrlSynth under a fixed number of data samples, and we have shown that our approach can enhance model performance compared to training solely on the original dataset. We agree that the efficiency of CtrlSynth is constrained by the resources required for image synthesis, particularly as datasets grow larger. Nonetheless, our method provides a valuable step forward within the existing setting, and future work could explore optimizations or alternative strategies to address these scalability concerns.

Can the authors provide details on the overall efficiency of the proposed pipeline? For example, how long does it take to generate 1 million images along with their captions? It would also be good to know the time cost of each component, e.g. vision tagging, caption generation, image generation. A more complete picture of the efficiency in the pipeline would better help to assess the value of this work.

Response 2: we will include detailed efficiency information in the revised paper. Below is a summary of the GPU hours (using H100) required for processing 1 million images and captions:

• Visual tagging: 89 GPU hours (52 for Florence, 16 for CatLIP, 21 for Qwn2 extraction)
• Caption generation: 32 GPU hours for running Mistral-Nemo inference
• Image generation: 4608 GPU hours for running SDXL

The datasets used (CC3M and CC12M) are relatively small, with no experiments conducted on larger datasets such as LAION-400M or LAION-5B.

Response 3: The main goal of CtrlSynth is to demonstrate the effectiveness and controllability of diverse text-image synthesis across different settings, including image-text datasets like CC3M and CC12M, as well as vision longtail datasets. We acknowledge that our dataset scale is relatively small, but scaling the synthesis pipeline to larger datasets would require substantial computational resources, particularly for image generation.

Moreover, we are unable to utilize the LAION datasets due to the presence of sensitive and NSFW content, as highlighted in recent research ("Into the LAION’s Den: Investigating Hate in Multimodal Datasets," https://arxiv.org/abs/2311.03449). This poses legal challenges that prevent us from using these datasets. Given our computing limitations and the short rebuttal timeframe, we cannot provide additional experimental results at this time. However, we plan to include experiments with DataComp1B for caption synthesis in the next version of our work.

The paper only tests a limited range of multimodal model structures, lacking experiments on models like BLIP and CLIP of different ViT models.

Response 4: Due to time and computational constraints, we will include experiments with CLIP using different ViT backbones (ViT-L and ViT-H) in the next version. For vision-language models (VLMs) such as BLIP and LLaVA, the experimental setup is more complex, as it requires two distinct stages of training: pretraining with image-text pairs and finetuning with instruction-tuning data (e.g., visual QA pairs). In future work, we will explore the impact of augmenting the pretraining image-text dataset with synthetic pairs generated by our CtrlSynth pipeline to better understand the benefits of our approach.

The study does not address data-efficiency validation. Existing data-efficiency-focused works, such as SemDeDup[2], Filter-&-Align[3], and Sieve[4], refine or filter datasets for better performance. The paper should include comparisons with these approaches in terms of model performance and the amount of training data.

Response 5: We appreciate the reviewer for highlighting related work on data efficiency. Our primary contribution is to generate diverse synthetic text-image data in a controlled manner, with data efficiency being an additional benefit of our approach. As illustrated in Figure 5 in Section 4.4, our synthetic samples demonstrate significant efficiency gains.

While our method is orthogonal to prior work on data efficiency, we have added a discussion in Section 4.4 of the revised paper to acknowledge and contextualize this relationship. Additionally, our synthetic samples notably improve performance on long-tail tasks, where conventional data filtering methods do not apply.

The proposed pipeline shares significant similarities with GenArtist[1] on image editing. The paper does not clearly demonstrate the differences between this work and GenArtist. It is important for the authors to specify these distinctions and highlight the novelty of their approach. Additionally, a thorough comparison should be incorporated into the experimental section to strengthen the evaluation.

Response 1: We appreciate the reviewer’s mention of the GenArtist paper. However, we would like to point out that GenArtist is a concurrent work, classified as contemporaneous according to the ICLR policy (refer to the FAQ section in the ICLR Reviewer Guide, https://iclr.cc/Conferences/2025/ReviewerGuide), and authors are not required to compare their work to the paper.

That said, we have clarified the key distinctions between our work and GenArtist. Specifically: (1) Our CtrlSynth pipeline offers support not only for controllable image synthesis but also for diverse text and image synthesis paths. In Section 3.2, we detail four unique synthesis paths that our method facilitates. (2) While GenArtist focuses on enabling more fine-grained and precise control over image generation, primarily benchmarking against methods like MagicBrush and InstructPix2Pix, our pipeline could benefit from such advancements. Nonetheless, it remains an open research question how to automate the generation of image editing instructions for each dataset sample, given that approaches like MagicBrush and InstructPix2Pix require manually crafted per-sample instructions and require additional training of text-to-image models to support such precise control.

While fine-grained control is presented as the main contribution, represented by the text and image controllers in the pipeline, the design is inadequate and lacks clarity. The design of the pipeline does not effectively demonstrate how the editing condition is provided to the generative model in a fine-grained manner. The text controller relies solely on prompt concatenation, making the mapping between visual tags and policies unclear and limiting precise control. Additionally, the paper does not address how to maintain image consistency after editing, which is essential for practical use. These shortcomings contribute to potential inconsistencies and an insufficient explanation of how fine-grained control is maintained. The image controller exists the same problem.

Response 2:
We have detailed our design approach in Section 3.1, where we outline the visual tags that enable users to exercise fine-grained control through editing. In line 201 of the revised version, we introduce three control policies for the text controller and two control policies for the image controller. It’s important to clarify that we do not modify the underlying models (LLMs or text-to-image models). Instead, our editing is facilitated solely through input textual instructions. While methods like InstructPix2Pix could be explored to maintain image consistency post-editing, applying such techniques at scale remains a significant challenge. Our primary objective is to augment existing datasets by synthesizing diverse samples. Interestingly, we find value in synthetic images that are not perfectly aligned, as they offer beneficial semantic augmentation to the original datasets. In fact, enforcing complete alignment in training can be detrimental, as it restricts the augmentation potential and hampers overall performance.

They say that the language model takes an instruction on how to generate a caption given the visual tags. They show some examples in Appendix A1. The instructions don't mention any editing, it mostly just says to describe the image better. In that case, do the gains come from some hallucination in the LLM caption that makes varied images?

Response 1: We appreciate the reviewer’s feedback and have provided additional clarification in our revised version.

Specifically, examples of text and image synthesis instruction templates can be found in Appendix A1. To be clear, our approach does not involve asking the models to perform editing tasks, nor do our instructions include editing steps, as these would be highly dependent on specific use cases. Instead, we explain that users have the option to modify visual tags manually (e.g., by adding or removing tags) and subsequently incorporate these edited tags into the instructions. The automation of editing instructions is beyond the current scope and is earmarked for future work. The performance improvements we report are derived from the model's ability to generate novel combinations of visual tags. This capability, which we refer to as semantic augmentation, is an intentional feature of our method. We have updated the text in Section 3.2 (line 219) to better articulate this point.

Have the authors tried any other variation of editing instructions? Is there any analysis on the kinds of image editing prompted by the text that improve performance more? Are there specific prompts that serve as better negatives when tuning the CLIP contrastive loss?

Response 2: To clarify, our CtrlSynth pipeline is designed to support various types of editing instructions in a plug-and-play manner. We have empirically observed that different kinds of image editing instructions yield similar performance outcomes. However, our synthetic pipeline is highly extensible and can readily accommodate more sophisticated or optimized instructions, should future research provide a more systematic evaluation or improved methodologies. Additionally, we did not experiment with enhanced negative samples for CLIP training. Our primary objective was to demonstrate the sample efficiency of our approach, and therefore, we adhered to the original baseline setup used in CLIP training to ensure a fair comparison.

There are other works that edit images based on text instructions like instruct pic to pic, magic brush etc. It might have been nice to see to see if editing certain things in images based on the LLM prompts is better than just using SD to generate since SD can often lack accuracy in generating the correct attribute object relation compositions.

Response 3: Thank you for highlighting the relevant papers. We have revised Section 2 (line 124) to include a discussion of InstructPix2Pix and MagicBrush in the related work. Prior image editing methods, such as InstructPix2Pix and MagicBrush, contribute valuable techniques and datasets aimed at enabling precise control over image generation. While our image synthesis approach could certainly benefit from these advancements, our primary focus remains on enabling diverse data synthesis. We acknowledge that automatically generating image editing instructions for each sample in a dataset is an open research question, which we hope future work will address.

Nit: There are several works that either generate synthetic images based on the dataset they want to target (https://arxiv.org/pdf/2406.05184), or for cross domain retrieval (https://arxiv.org/pdf/2401.00420). A discussion for comparison could be nice.

Response 4: Thanks for pointing out the related papers. We added the work in the related work section. Our pipeline can also be combined with previous work (https://arxiv.org/pdf/2406.05184) to improve the performance of cross-domain retrieval tasks or when the target task has little real data to retrieve (https://arxiv.org/pdf/2401.00420).

Thank you for the response. After reading the rebuttal, it seems to me that most of their performance gain comes from having more diverse data which is generated by their pipeline. However, the augemntation is on a smaller dataset. Since the gains don't come from generating any specific aspect which is hard to get in real data (like specific compositions that are hard to find in real images etc), a concern still remains whether simply using more real data is just easier than generating. Especially because their generation also seems to take a long time. For instance, if we look at some of the CLIP performance numbers when simply trained on a bigger dataset (eg, here: https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_results.csv), the performance is higher than their synthetic data augmented numbers. It would be nice if the authors could show the value of their synthetic data added vs just retrieving a similar number of examples more from the real data already available in bigger training sets.

However, that said, it is still interesting that synthetic generation from simply variations in captions can boost performance and hence, I keep my score. But I urge the authors to add such an experiment or acknowledge these in the limitations.