Beyond Fine-Tuning: A Systematic Study of Sampling Techniques in Personalized Image Generation

Thank you to the reviewer for pointing out the weaknesses.

Responses to specific questions are as follows:

1. While mixed sampling, stage sampling, multi-stage sampling, and mask sampling are generally straightforward to understand, Equation (11) appears to be merely a combination of the previous methods, making it somewhat complex and less engaging. The introduction of excessive hyperparameters may reduce its practical applicability. Incorporating a complementary adaptive hyperparameter selection strategy could significantly enhance its effectiveness.

We provide Equation (11) as an aggregation of the prior methods under the unified framework. We agree that generally speaking such combination has multiple independent hyperparameters. However, our main takeaway is that all those hyperparameters are unnecessary, and simple Mixed Sampling is optimal in terms of complexity and results. There is only one independent hyperparameter in the Mixed Sampling that should be user-defined as it controls the balance between concept fidelity and editability of the model. It should be noted that one can use Profusion to improve IS without compromising TS at the cost of doubling the computational budget.

2. Some of the experimental settings require more detailed explanations. Specifically, please clarify what 'TS' and 'IS' represent and elaborate on the necessity of using the Pareto Frontier in your analysis.

We use standard metrics to assess the quality of the personalized model. Let's denote real images of the concept and images that are generated using prompt p as

Let's denote a "clean" prompt (i.e., prompt without superclass name and placeholder token) as $\hat{p}$. Then IS is defined as:

And TS is defined as:

Where CLIP-I and CLIP-T denote CLIP image and text embeddings.

The Pareto Frontier appears naturally as a balance between concept fidelity and editability. We use that notion to show that different methods, despite their differences, usually lie below or on the curve that is defined by Mixed Sampling.

3. The purple dog in Fig. 1(b) is too similar to the concept dog. It is recommended to replace it with a dog that better represents the superclass to more effectively illustrate the distinctions between concept and superclass trajectories.

Thank you for you recommendation, we attached an updated version of the Figure 1 in Appendix K.

4. Typos: Page 5 line 259 "speciall" should be "special", Page 13, line 674: "Textual Inveersion" should be "Textual Inversion"; Page 2, line 078: "...by fine-tuning strategies; ;" should be "...by fine-tuning strategies;".

Thank you for your comment; we will correct these typos in the next version of the paper.

5. Fig. 2 shows the results of "a purple V*" and "A purple dog". Are there any changes in generating samples with "a purple V* dog"?

We need to clarify the exact way to generate images using different models. ELITE and Textual Inversion use "a purple V*" to generate samples, while DreamBooth, Custom Diffusion, and SVDDiff use "a purple V* dog". We simplify our notation by using the same prompt "a purple V*" across all models, even if the generation uses a superclass name inside the prompt. It's worth noting that this difference doesn't affect the way the Text Similarity is evaluated. We use the "empty" prompt without a superclass name or placeholder token (i.e., "a purple") to compute CLIP-T embeddings that are used to compute the metric.

Dear Reviewer,

Thank you for your valuable feedback and the time you have invested in reviewing our submission. If you have any additional questions, concerns, or need further clarification, we would be happy to continue the discussion.

Thank you once again for your attention. We look forward to your response.

Thank you to the reviewer for pointing out the weaknesses.

Responses to specific questions are as follows:

1. While the ablation is comprehensive the results are not necessarily surprising and it is also not clear to me what to do with the results. Different sampling approaches have different advantages and trade-offs (both in quality and sampling cost) and there is no winner. Rather, the best sampling approach is dependent on the model, desired outcome, and available inference compute. But I feel like that was clear before this and is also evidenced by the large related literation on sampling approaches for diffusion models, all of which have different trade-offs and advantages. So it's not clear what novel knowledge this ablation concretely contributes or how it would change the approach to current personalization approaches.

While there is a wealth of literature on sampling methods for diffusion models, their effectiveness in personalized generation is not thoroughly examined. A few studies propose test-time sampling, frequently merging it with fine-tuning approaches, model configurations, and fixed hyperparameters (ProFusion, Photoswap). However, these studies often overlook detailed ablation analyses that explain the contributions made by sampling and its distinguished components and the selection of hyperparameters.

As a result of our work, we can identify exactly what differences sampling methods have and what kind of improvement we can expect using them. Moreover, we observe that those insights can be extended to various finetuning methods like TI, DB, CD, and ELITE (see Figure 3, Appendix G) and architectures (see Appendix J).

2. What is the concrete outcome of the ablations? Do you suggest to use a default sampling approach or specific hyperparameters? Will that recommendation translate to other models besides SVDDiff?

The main outcome of the ablation is that we establish Mixed Sampling as the cheapest and most efficient way to balance concept fidelity with concept editability. We suggest using Mixed Sampling in cases where higher TS is required and Profusion when a higher IS is necessary. This result can be translated to other finetuning methods (see Appendix G) and to other architectures (see Appendix J).

We aim to encourage researchers to benchmark their work against naive techniques such as Mixing, ensuring that the complexity of newly proposed sampling strategies is well justified.

Dear Reviewer,

Thank you for your valuable feedback and the time you have invested in reviewing our submission. If you have any additional questions, concerns, or need further clarification, we would be happy to continue the discussion.

Thank you once again for your attention. We look forward to your response.

Thank you to the reviewer for pointing out the weaknesses.

Responses to specific questions are as follows:

1. The main concern I have about this paper is that it doesn’t convince the reader that test-time sampling strategies are effective and necessary for optimal personalization results. In particular, altering the sampling strategy at test-time indicates the personalization method used is not optimal, otherwise, the model should retain its capabilities in generating images from novel prompts. The gain using different sampling strategies is not significant, at least from the qualitative and quantitative results.

Despite the current personalization methods achieving high results, a common issue is that the balance between concept identity and prompt alignment can become disrupted in certain cases [1, 2]. While new personalization techniques aim to establish a more optimal balance, the example of SVDDiff illustrates that even an optimal position in the TS/IS plane does not guarantee satisfactory results with base sampling (see Figure 6). In such cases, advanced sampling strategies can effectively enhance these samples without requiring model retraining. To further substantiate this claim, we provide additional examples in which base sampling fails to align with the text, while more advanced sampling techniques significantly improve the outcome (see Appendix, Figure 12).

[1] CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization. Feize Wu, Yun Pang, Junyi Zhang, Lianyu Pang, Jian Yin, Baoquan Zhao, Qing Li, Xudong Mao.

[2] PALP: Prompt Aligned Personalization of Text-to-Image Models. Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, Ariel Shamir.

2. While the experimental section is well carried - it lacks deeper analysis beyond CLIP and Image similarity report.

We agree that more elaborate metrics for personalized generation exist; however, CLIP Image Similarity and Text Similarity are the most widespread. We complement CLIP Image Similarity with DINO Image Similarity in Appendix I (DINO Image Similarity) but find practically no differences.

Moreover, we provide a User Study that supports our findings that Mixed Sampling is the strong baseline for personalized generation, outperforming some of the more complex methods like Photoswap using the same or smaller computational budget.

3. The authors should consider new state-of-the-art personalization methods and compare the effectiveness of using test-time sampling strategies on newer methods.

In our experiments, we utilized core personalization models with open-source code to demonstrate that test-time sampling methods are independent of the fine-tuning approach and can be successfully transferred across various parameterizations (Figure 3). If you have specific methods you would like us to include in our experiments, can you please specify them, we will do our best to fulfill your requests.

I will raise my score to (6) - I think the paper is well written, the experiments are thorough and comprehensive, which gives a good insight about the trade-off between test-time sampling methods. Nonetheless, I think the paper lacks deeper insights.

Re 1. Please add the missing references in the paper - seems like they are relevant.

Re 2. I am aware that CLIP And Image Similarity metrics are the standard in this domain. My take-a-way message from the paper is that it is useful to use Mixing strategy to improve text-alignment of the personalisations methods. This finding is not surprising, and it has been proposed before (see [1] for example). Nonetheless, I do appreciate the authors efforts in doing thorough comparison between different methods. The authors are encouraged to provide deeper analysis in a follow-up/revised version - some examples for questions I have regarding test-time sampling strategies:

• Can you quantify the dis-similarity between the super-class and personalized- trajectories ? Even for the simple case-study, how the trajectories for the prompt="A photo of [V]" is different than that for the prompt="A photo of [Super-class]". Clearly the more they trajectories are different, the more likely the model overfits the training images. Will the Mixing strategy still work in the extreme case where the two trajectories are different (e.g., if the personalized model overfits the input data).

[1] https://arxiv.org/abs/2307.06925

Re 4: Thanks for taking the time to do the additional experiments.

Re 5: Thank you for justifying your choice.

Thank you for raising the score. We appreciate that you acknowledged our thorough and comprehensive experiments and found our paper well-written.

Regarding the dissimilarity of the trajectories, Figure 3 illustrates the full plots for mixing sampling, including the border points. On the far left, we show the sampling using only the concept, while on the far right, we use only the superclass for sampling. We observe that the differences between the trajectories are strongly model-dependent. For instance, in the case of Textual Inversion, where the model remains unchanged and only the embedding is trained, this difference is significant. In contrast, when we alter the model weights, the difference diminishes.

Significant overfitting can reduce the pronounced effect of sampling. On the other hand, blending with the trajectory of a non fine-tuned model may significantly expand the Pareto frontier; however, it might introduce artefacts during the blending process due to the substantial differences in trajectories.

We will take your comments into account and endeavour to carefully investigate these effects in future versions.

Thank you to the reviewer for pointing out the weaknesses.

Responses to specific questions are as follows:

1. The main weakness is in the contributions. Although this manuscript presents a comprehensive study of various sampling strategies, they are all based on existing techniques. There are not many new ideas presented in the manuscript.

Even though the techniques presented in the paper already exist, there is no literature providing their fair comparison in the general formulation. Furthermore, these techniques are often presented in complex combinations with others (such as the refining step utilized in ProFusion or the self-, cross- attention map replacement featured in Photoswap) and varying fine-tuning strategies.

In response to this gap, our research offers a more fair comparison by standardizing a single fine-tuning parameterization across all methods. We also decompose each sampling strategy, breaking it down into less complicated components. Our findings reveal that basic techniques like Mixing and Switching can serve as strong baselines that are effective and often sufficient for addressing text misalignment while maintaining concept identity. This analytical approach stands as a noteworthy contribution to the existing body of knowledge.

Our study aims to encourage researchers to benchmark their work against these naive techniques, ensuring that the complexity of newly proposed sampling strategies is well justified. Additionally, we advocate for fair comparisons by standardizing fine-tuning methods.

2. The results shown in the paper are all easy cases with common objects. It is not sure if the conclusion will stay the same for more challenging cases involving unique objects.

We selected the Dreambooth dataset for its widespread use in the field which helps to contextualize our results with the prior works. Some concepts represent unique objects like can or rc\_car that should be challenging enough to find in the generated images of the source model. Can you specify what classes you think is important to analyze?

We agree that it is important to check whether our findings hold for the challenging prompts. We provide such analysis in Appendix H (Complex Prompts Setting). To summarize, we found the same behavior of the sampling methods in this case.

3. All experiments are based on a single diffusion model backbone (Stable diffusion 2) and fine-tuning method SVDiff. It is not sure if the conclusion will still hold for other backbones and finetuning methods

As for the finetuning methods, we have already analyzed some methods like Textual Inversion, Dreambooth, Custom Diffusion, and ELITE in combination with Mixed sampling in our paper (see Figure 3 in the paper). However, to fill the gap we provide an analysis of other samplings in combination with DreamBooth in Appendix G (Dreambooth Results) and show that our results hold for more simple methods than SVDDiff.

Also, we provide results (Appendix J) for the Stable Diffusion XL and Pix-Art to show that even completely different architectures like Visual Transformer still follow the same patterns as the simple and easy-to-train Stable Diffusion 2.0.

4. Would these sampling strategies also be helpful for non-finetuning personalized text-to-image diffusion models, for example ELITE?

In Figure 3, we present the results of applying mixed sampling to ELITE. The overall impact on the metrics is similar for this model as it is for fine-tuning-based models.

Dear Reviewer,

Thank you for your valuable feedback and the time you have invested in reviewing our submission. If you have any additional questions, concerns, or need further clarification, we would be happy to continue the discussion.

Thank you once again for your attention. We look forward to your response.