Increasing the Utility of Synthetic Images through Chamfer Guidance

We would like to clarify the models used in our experiments:

• LDM1.5 refers to Stable Diffusion 1.5, a widely-used, U-Net-based implementation with 860 million parameters.
• LDM3.5M refers to Stable Diffusion 3.5 Medium, a flow-based Multimodal Diffusion Transformer with 2.5 billion parameters representing state-of-the-art among openly available generative models.

We then thank the reviewer for finding that our work tackles an important question in the literature, and that a key strength compared to c-VSG is the reduced computational complexity of using the Chamfer distance.

[W1] Novelty

We believe our work presents a significant contribution, as highlighted by other reviewers (a novel training-free guidance method that effectively enhances synthetic image utility, and that the idea of using Chamfer distance as guidance reward is conceptually elegant).

Our primary contribution is a simple, effective, and computationally efficient In-Context Learning (ICL) framework for generative models. While ICL has transformed language modeling, its application to steer generative vision models is significantly under-explored. We are among the first to propose a practical method for this purpose.

The novelty lies in creatively adapting the Chamfer distance, traditionally used for point clouds, as a guidance signal within a diffusion model feature space. This introduces a new way to control generation based on few-shot examples.

We demonstrate state-of-the-art results in the downstream usefulness of generated synthetic data with minimal data. These constitute a substantial contribution.

[W2] Access to real data

We argue that our work addresses a well-established and increasingly critical problem in generative modeling.

The problem of adapting a generative model to a target distribution given a few reference examples is a significant area of research, which is foundational to several applications:

• Few-shot Adaptation: This is a common setting for tasks like data synthesis for imbalanced classification, as explored in prior work [1, 2], which we also include as baselines.
• Personalization: The goal of personalizing large text-to-image models for subject-driven generation, as seen in impactful methods such as Textual Inversion [3], DreamBooth [4], and SVDiff [5].

More broadly, our work is an instance of In-Context Learning (ICL) for diffusion models. ICL is a cornerstone of modern LLMs [6], and its application to generative vision models is a critical emerging direction [7].

We are among the first to successfully adapt large diffusion models via ICL without requiring finetuning.

[W3,Q3] Scalability and extension to text-to-image

Our work focuses on object-centric and class-conditional generation, aiming to improve the downstream utility of synthetic data (e.g., for classification and geographic representation). This is a well-established research area, distinct from open-ended text-to-image synthesis, which is a different scope. Our contribution is a novel guidance mechanism that excels within this paradigm.

However, extending our method to text-to-image is a promising future direction. One approach could be: Step 1. Offline retrieval database: Embed a large text-image dataset into semantic vectors based on their captions. Step 2. Inference-time Retrieval: For a user's text query, retrieve the top-k relevant image-text pairs and use their images as exemplars for our guidance.

We perform an early experiment by implementing this using COCO. For Step 1, we use the image-text pairs from the COCO training dataset. We embed the captions with CLIP, and we use FAISS for fast retrieval. For Step 2, we subsample 5000 captions from the validation set and retrieve the top-k closest images to measure the Chamfer distance over. In this setting, we compared the performance of LDM1.5 with default sampling and our Chamfer Guidance. Results are reported in the table below, and we can observe that our method brings substantial improvement in this setting as well.

[W4] Additional baselines

We initially omitted them as they address orthogonal research goals. Those methods increase ungrounded diversity without specific targets, similarly to CADS, which we employed as a baseline, while our work focuses on grounded generation.

We have implemented these methods. As shown below, while they may increase ungrounded diversity (recall), they don't perform well on our grounded diversity task requiring fidelity to exemplars.

[Q1] Why Chamfer distance

Our decision was based on its combination of theoretical and practical advantages of the Chamfer distance for our specific task.

Chamfer distance is a well-established and principled metric for comparing point clouds, which is how we represent features in the embedding space. Its two components naturally map to the desirable properties of fidelity and diversity in generative modeling. It is also computationally efficient, as it relies on a simple 1-Nearest Neighbor (1-NN) search. This efficiency is critical for its usage as a guidance signal in the diffusion sampling loop.

We also considered other candidate optimization targets but found them unsuitable:

• Fréchet Inception Distance (FID): It was not feasible. It requires large distributions of both real and generated data to be meaningful. Moreover, its computational cost would make it prohibitive to optimize at each sampling step.
• Optimizing Recall/Coverage: These metrics also require access to the full data distributions for accurate estimation. More importantly, because they involve counting, they are non-differentiable, posing a significant optimization challenge.

[Q2] Different CFG scales

This analysis was included in our original submission (Table 4 and Table 10 in Appendix). Results demonstrate our method's improvements are consistent across CFG scales. For the reviewer's convenience, we are reporting these key results below.

[Q4] Multi-prompt scenario

While our current focus is on object-centric generation, tackling text-to-image, for which we showed preliminary results in [Q3], and in particular multi-prompt applications would be an interesting future direction. We interpret "multi-prompt" as using a single set of visual exemplars to guide generation for several different text prompts.

Our approach would suit this task because our guidance operates in the feature space of the exemplars. This effectively decouples the visual concept (the exemplars) from the semantic context (the prompt).

Our guidance would ensure the generated concept is consistent with the exemplars, while the context and style are dictated by the specific prompt. Evaluating such a use case would require suitable data and parameter tuning.

[Q5] Large diffusion models

Our experiments include LDM3.5M, the 2.5 billion parameter Stable Diffusion 3.5 Medium model. This is a state-of-the-art Multimodal Diffusion Transformer, competitive with other large models like SDXL and FLUX. Thus, our method has been validated on current, large-scale diffusion models.

---

[1] Hemmat, R. A., et al. Feedback-guided Data Synthesis for Imbalanced Classification TMLR 2024
[2] Hemmat, R. A., et al. Improving geo-diversity of generated images with contextualized vendi score guidance ECCV 2024
[3] Gal, R., et al. An image is worth one word: Personalizing text-to-image generation using textual inversion ICLR 2023
[4] Ruiz, N., et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation CVPR 2023
[5] Han, L., et al. Svdiff: Compact parameter space for diffusion fine-tuning CVPR 2023
[6] Donq Q., et al. A Survey on In-context Learning EMNLP 2024
[7] Wang, Z., et al. In-context learning unlocked for diffusion models NeurIPS 2023

Dear Reviewer wmJJ,

I notice that the authors have submitted a rebuttal. Could you please let me know if the rebuttal addresses your concerns? Your engagement is crucial to this work.

Thanks for your contribution to our community.

Your AC

Extensibility to Text-to-Image Generation:

We appreciate that the reviewer found our initial text-to-image results on COCO to be promising. Our primary objective for the rebuttal was to empirically demonstrate the feasibility and potential of extending our framework to this domain. We will include the current text-to-image results in the camera-ready Appendix with qualitative examples.

Multi-prompt applications

While we acknowledge the reviewer's point on multi-prompt applications, we respectfully disagree on its role as a key measure of generalizability and applicability. In our manuscript, we showed generalizability and applicability of our approach by reporting the following results: 1) object-centric representation, 2) mitigating representational biases in the "Geographic Diversity" setting, and 3) improving the downstream performance of a classification model trained on our generated data. In the rebuttal, we added results for COCO to prove the point that our method is applicable to more complex text conditionings, and provided further validation on the few-shot adaptation of the generative model in the context of increasing the utility of synthetic data. Chamfer guidance consistently outperformed prior art, proving its generalizability to multiple validation scenarios.

We would also like to clarify that our work already includes experiments that function as a multi-prompt evaluation. Specifically, our geographic diversity experiments on GeoDE and DollarStreet operate on this principle. In that setup (following cVSG), we use a fixed set of visual exemplars for a single object concept while varying the context within the text prompt (e.g., generating "a {object} in {region}"). This effectively tests the model's ability to apply a consistent visual concept across different textual contexts, which is the core challenge of the multi-prompt scenario the reviewer describes. The results, presented in Table 2 of our manuscript, demonstrate the success of our method in this task. Our Chamfer guidance achieves superior distributional coverage and better prompt adherence (measured by CLIPScore) when compared to all baselines, especially when using CLIP as the feature extractor.

Efficiency and Feature Extractors bias:

We wish to clarify the concerns regarding computational overhead and feature extractor bias.

Computational Efficiency: The overhead of our method is taken into account. On top of the efficiency analysis in Appendix A, we would like to highlight that our Chamfer Guidance is applied sparsely (once every 5 steps, as noted on L217), following the implementation of c-VSG. This design choice drastically reduces the computational load compared to guidance at every step, leading to a 31% reduction of FLOPS than CFG for LDM3.5M.

Bias: Our method improves representativeness of regions as our experiments on GeoDE and DollarStreet show (Tables 2, 5, 6, 7, and 8). We are consistently improving the worst-region performance across all metrics, and we achieve this with both DINOv2 and CLIP as feature extractors, thus reducing representational bias.

We thank the reviewer for highlighting that ours is a novel inference-time method to guide image generation using a small set of real exemplar images, that delivers a clear advance in generation utility, yielding simultaneously higher diversity and fidelity than strong baselines, and that the idea of using Chamfer distance as a guidance reward is a conceptually elegant alternative to previous approaches.

[W1,2 Q2] Sample selection and stochastic process

Regarding the sample selection, the reference samples were chosen at random from the dataset without a specific sampling strategy. This approach is standard and consistent with prior works like c-VSG [2]. While exploring advanced sampling techniques is interesting for future research, our focus is on the novel generation method itself. Our experiments confirm that even with random sampling, our method produces high-quality, useful images. To provide empirical evidence, we ran an additional experiment on the GeoDE dataset using two different random seeds to select exemplar images. As the table below demonstrates, the standard deviation across runs is negligible.

Concerning the stochastic nature of the diffusion process, we mitigate this through the scale of our experiments. By generating large numbers of images (e.g., 50,000 for ImageNet-1k), we ensure robust metrics that represent stable averages.

[W3] Specialized use case

We respectfully disagree that our work is a "specialized use case." Our work addresses a well-established and critical problem in generative modeling.

The problem of adapting a generative model to a target distribution given few reference examples is a significant problem. This setting is foundational to:

• Few-shot Adaptation: Common for tasks like data synthesis for imbalanced classification [1, 2]
• Personalization: Personalizing large text-to-image models for subject-driven generation, as in Textual Inversion [3], DreamBooth [4], and SVDiff [5]

More broadly, our work is an instance of In-Context Learning (ICL) for diffusion models. ICL is a cornerstone of modern LLMs [6], and its application to generative vision models is a critical emerging direction [7].

Our primary contribution is an approach that is both effective and highly efficient. We are among the first to successfully adapt large diffusion models via ICL without requiring any model finetuning, achieving state-of-the-art results with few reference samples.

[W4] Computational overhead

Our Chamfer Guidance can be significantly more computationally efficient than standard sampling with Classifier-Free Guidance (CFG), as analyzed in Appendix A.

Standard CFG sampling requires a double forward pass through the model at each diffusion step, doubling the floating-point operations (FLOPS). When applied to conditional-only models (CFG scale = 1), our guidance mechanism is more efficient. With the LDM3.5M model, our method achieves up to a 30% reduction in FLOPS while reaching state-of-the-art results.

[W5] Feature extractor

For the “Geographic Diversity” setting (Table 2, and Table 6 and 7 in the Appendix), we explored DINOv2 and CLIP as feature spaces for our guidance. Both lead to state-of-the-art results over prior works, with DINOv2 leading to better results than CLIP.

We then chose DINOv2 also for the other settings because of its self-supervised learning pre-training, which learns rich, instance-level visual features more suitable for our guidance task than features from classification models (Inception) or image-text alignment models (CLIP).

We validated this choice empirically by testing DINOv2 and CLIP on the GeoDE dataset. Our findings confirm that DINOv2 provides superior performance, with CLIP bringing improvements w.r.t. baselines as well.

[W6] Zero-shot scenario

Our method is designed for a few-shot setting with reference data to guide generation, related to In-Context Learning (ICL). However, we find extending to a data-free pipeline interesting for future work. We could envision a self-bootstrapping technique:

1. Generate initial candidate images for a given class
2. Automatically select a diverse subset maximizing "diameter" (coverage) in a robust feature space
3. Use this synthetically-generated set as guidance exemplars

[Q1] Chamfer distance choice

Our decision was based on theoretical and practical advantages. The Chamfer distance is well-established for comparing point clouds, which is how we decided to represent images, with components naturally mapping to fidelity and diversity in generative modeling. It is computationally efficient, relying on simple 1-NN search, critical for guidance in the diffusion sampling loop.

We considered other targets but found them unsuitable:

• FID: Not feasible as it requires the full distribution to be stable, and it is computationally prohibitive to backpropagate through.
• Recall/Coverage: They require full data distributions and are non-differentiable.

[Q3] Shifted distribution

Our geographic diversity study used GeoDE and DollarStreet datasets, known to be challenging with distributions shifted from generative model training data. Base models struggle with these specialized datasets [2, 8].

Our results demonstrate that Chamfer Guidance is robust and effective in this setting, successfully steering generation to align with few-shot exemplars from shifted domains, proving practical utility in real-world scenarios.

[Q4] Guidance scale

The CFG scale for each model was chosen by performing a sweep and selecting the value maximizing F1 score on a validation set. For the large LDM3.5M model, we chose $\omega = 1$ also to speed up generation, given the size of the model.

---

[1] Hemmat, R. A., Pezeshki, M., Bordes, F., Drozdzal, M., & Romero-Soriano, A. Feedback-guided Data Synthesis for Imbalanced Classification. TMLR 2024

[2] Hemmat, R. A., Hall, M., Sun, A., Ross, C., Drozdzal, M., & Romero-Soriano, A. Improving geo-diversity of generated images with contextualized vendi score guidance. ECCV 2024

[3] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. ICLR 2023

[4] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR 2023

[5] Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., & Yang, F. Svdiff: Compact parameter space for diffusion fine-tuning. CVPR 2023

[6] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A Survey on In-context Learning. EMNLP 2024

[7] Wang, Z., Jiang, Y., Lu, Y., He, P., Chen, W., Wang, Z., & Zhou, M. In-context learning unlocked for diffusion models. NeurIPS 2023

[8] Hall, M., Ross, C., Williams, A., Carion, N., Drozdzal, M., & Soriano, A. R. Dig in: Evaluating disparities in image generations with indicators for geographic diversity. TMLR 2024

We thank the reviewer for finding our work a novel training-free guidance method that effectively enhances the utility of synthetic images that achieves state-of-the-art performance across both object-centric and geo-diversity benchmarks while significantly reducing computational cost, and that theoretical motivation, practical utility, and clean experimental design make it a strong contribution to synthetic data generation.

[Q1] Embedding space

For the “Geographic Diversity” setting (Table 2, and Table 6 and 7 in the Appendix), we explored DINOv2 and CLIP as feature spaces for our guidance, as DINOv2 and CLIP feature extractors correlate better with human judgment of similarity than Inceptionv3 [1]. Both led to state-of-the-art results over prior works, with DINOv2 leading to better results than CLIP. To further validate our choice of DINOv2, we conducted a preliminary empirical study comparing its performance as a feature extractor for performing Chamfer guidance against CLIP on the GeoDE dataset in an "object-centric" setting, e.g., with prompts like “a photo of a car.”. These experiments were run using LDM1.5.

Our findings were as follows:

• DINOv2 consistently yielded the best performance, showing substantial gains in both diversity and fidelity. Similarity in DINOv2 space also correlates to human-perceived similarity slightly more than CLIP, as reported in [1].
• CLIP improved upon the baselines, and in particular coverage scales with the number of guiding samples. With higher k we observe reduced marginal improvements, and we deem this to the fact that the model tended to converge towards generating an average representation of the object. We hypothesize this is because CLIP's pre-training is "concept-centric" (aligning images to general text concepts), whereas DINOv2's is "instance-centric" due to its self-supervised training, making it better at preserving the unique features of a specific reference image. T We have included a summary of these results in the table below. This analysis confirmed that DINOv2 was the most effective choice for our method, with CLIP being an alternative. We will include these results in the final version of the manuscript.

[Q2] User Study

While the primary focus of our study is the utility of the generated data for representation and as a training source for downstream tasks, which is confirmed by our extensive experimental evaluation with downstream classifier training, we agree that human evaluation provides valuable insights.

To that end, we conducted a small-scale user study to complement our quantitative findings. We collected 965 data points from more than twenty annotators. In this study, users were presented with samples generated from prompts based on the ImageNet dataset. Their task was to choose their preferred generation in a side-by-side comparison between images from the base LDM3.5M model and images generated with our Chamfer Guidance applied to the same model. Users were also presented with real images coming from the dataset to ground their evaluation in real-world quality and coherence.

The results showed a strong preference for our method: images generated with our Chamfer Guidance were preferred in 92% ± 2% of the cases. This suggests that the automatic evaluation of quantitative improvements in downstream utility also correlates with enhanced human-perceived quality and fidelity to the target concept distribution. We will include these results and details of the study in the revised manuscript.

[Q3] $\gamma$ ablation

We would like to direct the reviewer to our detailed ablation study in Appendix E, where we analyze this point on the ImageNet1k dataset with LDM1.5. Our results show that our method is robust across different values of $\gamma$. In fact, for all tested strengths, our method surpasses the LDM1.5 baseline in terms of F1 score, demonstrating its consistent effectiveness.

Our analysis reveals a trade-off between guidance strength and perceptual quality as measured by FID:

• Optimal Performance: The configuration with $\gamma=0.07$ and k=32 achieves the best overall results, yielding the highest F1 score (0.931) and precision (0.950). This indicates that stronger guidance produces samples with increased fidelity and diversity.
• Performance Trade-off: While stronger guidance boosts precision and coverage, it can slightly degrade the FID. For example, increasing $\gamma$ from 0.05 (FID: 8.840) to 0.07 (FID: 13.670) shows this effect. The moderate guidance maintains a better balance between precision/coverage and overall image quality.

We hypothesize that this increase in FID with stronger guidance might be partially due to a metric mismatch. The FID metric is computed using features from the Inception network, which is optimized for classification. Our guidance, however, operates in the DINOv2 feature space, which is richer in instance-level semantics. This discrepancy may lead the Inception-based FID to penalize valid, high-fidelity generations that are perfectly aligned in the DINO space.

We report the table below.

—-

[1] Hall M., et al. Towards Geographic Inclusion in the Evaluation of Text-to-Image Models. In ACM FAccT 2024.

Dear Reviewer mRsF,

I notice that the authors have submitted a rebuttal. Could you please let me know if the rebuttal addresses your concerns? Your engagement is crucial to this work.

Thanks for your contribution to our community.

Your AC

Dear reviewer mRsF,

Thank you for your response and for taking the time to read our rebuttal. We are glad that our explanation has addressed your concerns. We appreciate you upgrading your rating.

We would be grateful if you could let us know which specific points of concern remain, as we would be happy to discuss them further during this period. Please let us know if there is anything else we can clarify.

Submission 20967 Authors