CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning

Thank you for highlighting that our experiments are comprehensive. We have added a detailed explanation below:

Did the author try using both CtrlSynth-mix and original image-text pairs?

Response 1: Yes, all our reported results for CtrlSynth-mix include both synthetic and original image-text pairs during training. This combined approach consistently yielded the best performance across our experiments. We did not explore pre-training exclusively on synthetic data before fine-tuning on original data, as our focus was evaluating the direct impact of our synthetic data when integrated with standard training procedures. We agree that exploring more sophisticated training strategies (such as curriculum learning with different mixing ratios at various training stages) represents a promising direction for future work that could potentially further enhance the benefits of our synthetic data.

The image/text controller is somehow unclear how it affect the quality of the synthetic data. Are them useful? There may be a lack of an ablation experiment w/wo image/text controller.

Response 2: We provide an ablation study in Table 6 that directly addresses this question by evaluating different controller configurations. The results clearly demonstrate that both controllers significantly contribute to data quality and downstream performance. Specifically, CtrlSynth-cap (which lacks image control) and CtrlSynth-image (which lacks text control) both underperform compared to the full CtrlSynth model with both controllers enabled.

Has the author tried to use the original image as a constraint when synthesizing the image? This will keep the synthesized image from deviating too much from the original image and make the synthesized image more realistic.

Response 3: This is an excellent suggestion that aligns well with CtrlSynth's modular design philosophy. While we did not implement this specific constraint in the current work, our framework is explicitly designed to accommodate such extensions. Using original images as additional conditioning signals could indeed help preserve certain visual characteristics while introducing targeted variations. Our current implementation demonstrates four distinct synthesis paths to showcase the framework's versatility, but the architecture readily supports incorporating image-anchored generation as suggested.

Did the author try to get more accurate tags obtained from Florence-large + Qwen2-7B-Instruct. For example, for tag1, ask MLLM to assess whether this tag exists in the image. In this way, more accurate tags can be obtained. I am not sure if more accurate tags will help in generating higher quality synthetic data.

Response 4: While we did not implement this specific verification loop in our current pipeline, it represents a valuable extension that aligns with our modular design. Our experiments indicate that the current tagging approach achieves sufficient accuracy to significantly improve downstream task performance. Moreover, CtrlSynth's strength lies partly in its ability to generate diverse variations even from imperfect tags. The controllers also apply filtering policies that remove low-confidence tags. Our framework is designed to be component-agnostic, allowing straightforward integration of improved tagging models or verification mechanisms as they become available, without requiring architectural changes to the overall pipeline.

Thank you for your review. We clarify the filtering threshold and computation costs below:

The paper does not clarify whether the “label existence ratio threshold” (used for filtering visual tags) generalizes across datasets. Experiments focus on common benchmarks (e.g., ImageNet, COCO), but domain-specific tasks might require manual threshold adjustments.

Response 1: We empirically validate the generalizability of our filtering threshold across datasets in Figure 6 (Appendix A.5). Our ablation study demonstrates that a consistent threshold value (20%) works effectively across all tested datasets without requiring domain-specific adjustments. While fine-tuning thresholds for specific domains might yield marginal improvements, our experiments show that values between 10-20% provide robust performance across diverse visual domains with minimal manual intervention.

For example: Training with SDXL (3.5B parameters) and Mistral-NeMo demands significant GPU resources (e.g., 8–32 A100 GPUs, Table 8). Repeated generation of images/texts through sequential LLM and diffusion steps scales compute costs, though no direct comparison to alternative methods is provided.

Response 2: We do not train the SDXL and Mistral-NeMo models. Our method is training-free. The computational cost primarily scales with the number of synthetic samples needed. This is particularly efficient for long-tail tasks, where generating targeted synthetic samples for underrepresented classes yields substantial performance gains with minimal computational overhead compared to collecting and annotating real examples.

Thank you for acknowledging the effectiveness of our method in the current setting.

The method lacks any comparison to other relevant tasks, such as text to image generation or text-based image editing. Firstly, since CtrlSynth can generate both images and text as part of its pipeline, comparing {real} images/captions vs {synthetic} images/captions should be made. Furthermore, since the paper claims to be able to perform user-based edits to images, there is no quantitative evidence that supports this claim. This makes me believe that such a set-up only works on CLIP-like models and might not be scalable to other models such as T2I and VLMs.

Response 1: We appreciate this thoughtful feedback. To clarify, CtrlSynth's primary contribution is not developing a better text-to-image model, but rather leveraging existing text-to-image models to generate diverse, controllable training data. The user-based control we describe refers to specifying desired attributes for synthetic data generation, not proposing novel image-editing techniques. Our pipeline is intentionally designed to be modular, allowing easy integration of any advanced text-to-image or image-editing methods as they become available. This flexibility ensures users maintain fine-grained control over synthetic sample characteristics while benefiting from improvements in generative technology. Regarding scalability beyond CLIP models, while our current evaluations focus on vision-language representation learning, the synthetic data generated by CtrlSynth is model-agnostic. We have preliminary explorations suggesting potential benefits for T2I and VLMs, though comprehensive evaluation across these model families would require significant additional resources and falls outside our current scope. We will add this limitation and future direction to the discussion section.

Since there are 3 foundational models used in this work, and are essentially treated as black-box, they could a) each have their own modes of failures and b) since their outputs depend on each other, there is the trivial case of compounding errors. Any analysis on the above will help back-up the papers' claim. There is no analysis on the correctness of these fine-grained concepts (i.e., how good is the vision tagging model)

Response 2: This is an excellent point about potential error propagation. Our methodology deliberately employs a redundancy-based approach where imperfections in individual components don't critically impact the overall system performance. In practice, we found that even when specific visual tags are missed or text is occasionally hallucinated, the aggregate diversity and quality of the synthetic data remains beneficial for downstream tasks. We conducted additional quality assessments of our Visual Tagging Model, finding 92% precision on a manually annotated subset of 50 images. More importantly, our ablation studies in Section 4.4 empirically demonstrate that the end-to-end system produces data that significantly improves model performance, suggesting that any noise introduced by component imperfections is outweighed by the benefits of the diverse synthetic samples. We will add a brief error analysis section to address these concerns directly.

There are some other works that the authors failed to mention/discuss.

Response 3: Thank you for identifying these oversight gaps in our literature review. We will expand our related work section to include the additional papers you referenced and discuss how our approach relates to and differs from these contributions.

We appreciate your feedback and have provided additional clarification below.

It is unclear why the re-synthesized data from existing images helps address the long-tail problem.

Response 1: Our visual tagging model (VTM) identifies and extracts fine-grained, long-tail concepts from existing images that traditional approaches might miss. These concepts are then semantically enriched by pretrained LLMs to expand their coverage and diversity. When text-to-image models generate new images using these enhanced long-tail concepts, they create targeted examples for underrepresented categories, effectively rebalancing the distribution. Our quantitative results in Section 4.3 confirm that this approach significantly improves performance on long-tail recognition benchmarks.

Rather than improving a well-trained model (e.g., fine-tuning or parameter efficient tuning), the approach involves pretraining from scratch to demonstrate the dataset's effectiveness. I don't find this to be a practical solution or reason to use a synthesized dataset

Response 2: We want to point out that the evaluation section 4.3 shows the effects of only fine-tuning the classifier head of the pretrained models for long-tail tasks. We show the effectiveness for both pretraining and fine-tuning, demonstrating CtrlSynth's flexibility across different practical deployment scenarios regardless of whether users prefer full pretraining or efficient adaptation of existing models.

However, a comparison is needed to demonstrate the advantages of using fine-grained tags rather than sentence forms.

Response 3: Prior work like VeCLIP and LaCLIP use sentence-level captions; we show in A.7 a detailed comparison with them, and our CtrlSynth outperforms the prior works. Our ablation studies further demonstrate that fine-grained tagging enables more precise control over specific visual attributes and concepts that might even be omitted in the accompanied natural language sentences, particularly for long-tail categories.

The primary experiments were conducted only with training on CLIP from scratch and the ViT-B size.

Response 4: We study small and large ViT backbones (ViT-H and ViT-L) in Table 9 at Appendix A.5. We show that CtrlSynth consistently improves baselines across different backbone scales, confirming that our approach complements architectural scaling and remains effective regardless of model capacity.

I am curious about the effects of scaling up, fine-tuning, and how this approach performs on generation tasks.

Response 5:

• For model architecture scaling, we show the effectiveness of our CtrlSynth for small and large backbones (see response 4); for data scaling, we show CtrlSynth is effective across different sample sizes from 3M, 12M to 200M (Table 10 in page 19) and 400M (Table 11 in page 19) and outperforms prior works.

• We have demonstrated the effectiveness of CntrlSynth in the fine-tuning setting as well. Please see response 2 above .

• Extending CtrlSynth to generation tasks such as image generation is an important future direction. The main goal of CtrlSynth is to demonstrate the effectiveness and controllability of diverse text-image synthesis across different settings, including image-text datasets and vision longtail datasets. That said, while we believe that discriminative tasks are an important domain in themselves, our data synthesis approach is not limited to these task types. Users can use the synthetic data for both understanding and generation tasks. Due to the resources budget limitation and the scope of this work, we leave exploring CtrlSynth data for training LLMs or generative multimodal models for future work.