ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation

This paper introduces the task of prompt-adaptive workflow generation for text-to-image models, addressing the complexity of creating effective workflows that combine multiple specialized components. The authors propose two LLM-based approaches: a tuning-based method that learns from user preferences, and a training-free method that selects existing workflows based on the prompt. Both methods improve image quality compared to monolithic models or prompt-independent workflows, highlighting the potential of prompt-dependent flow prediction as a new pathway for enhancing text-to-image generation.

This paper introduces a new approach to text-to-image generation by dynamically adapting workflows to input prompts, rather than using monolithic models. The authors present two methods for workflow generation: ComfyGen-IC (an in-context prompt classifier) and ComfyGen-FT (a fine-tuned model for predicting workflows based on prompt and desired image quality score). By leveraging ComfyUI, which stores workflows as JSON files, the model maps prompts to workflows. The evaluations based on preference estimation models and user study are performed to validate the superiority of the proposed pipeline over fixed workflows and single-model baselines.

This paper introduces a novel approach to text-to-image generation by developing adaptive workflows that cater to specific user prompts, enhancing the quality of generated images. The authors propose two Large Language Model (LLM)-based methods: ComfyGen-IC, which uses an in-context learning approach, and ComfyGen-FT, which involves fine-tuning an LLM. These methods are designed to automatically synthesize workflows that better match user prompts, improving the standard approach of using a single, monolithic model.

The paper considers image generation with workflow systems such as ComfyUI. The main issue when using the workflow systems is the challenge of developing a valid flow that would lead to high quality image generation. The suggested approach leverages a LLM to generate a prompt conditioned flow. In particular, the paper is exploring two approaches: finetuning and in-context learning. Automatic and human evals show that the proposed approaches are outperforming fixed flows as well as off-the-shelf models.

This paper proposes a prompt-adaptive workflow generation approach that utilizes LLMs to automate the workflows, thereby assisting users in their creative processes. The authors implement both retrieval-based and fine-tuning methods, building a comprehensive dataset consisting of <prompt, workflows, score>. The experiments demonstrate that the proposed method outperforms existing approaches. Additionally, a detailed user study is conducted, and the authors analyze the originality, diversity, and human-interpretable patterns of the method.

We would like to thank the reviewers for providing feedback on our manuscript.

We were glad to see agreement that the task and approach presented in the paper are innovative or valuable, with reviewers highlighting both the quality of the results and the potential impact on the creative industries.

While we are withdrawing our work, we will incorporate some of your suggestions into the next version. Here, we want to offer a general response to some of the concerns raised by the reviewers.

One key point we wish to highlight is that research is an ongoing effort. Indeed, multiple reviewers raised ideas through which the method could be improved, but we do not think this is a flaw. Our paper introduces a new task, and we demonstrate that there is value in pursuing this task by showing a solution that already improves over existing, relevant baselines.

The fact that there are multiple clear paths for future improvement simply means that there is a lot of potential in this new field, and that it is likely to spur future research. Both of these are traits which we argue should be a reason for acceptance to a scientific conference, rather than for rejection.

Below, we address a few specific points raised by the reviewers.

Limited output flow diversity

Our paper introduces a new task, and we present two methods that show the concrete benefits of tackling this task. The limited diversity is an artifact of our current solution, but we do not believe this is a fundamental limit to future solutions for the same task. Indeed, our work and some of the reviewers already suggest ways in which this could be overcome. Importantly, even at the current levels of diversity, our solutions meaningfully outperform all relevant baselines.

Inclusion of stronger models like SD3 or Flux.

At the time of our submission, both models were fairly recently released and did not yet have an extensive ecosystem of specialized variants such as fine-tuned versions, improved VAEs or LoRAs. This limits the flows involving them to very few changes in components, which we believe would make for a less interesting scenario to investigate. For SDXL, this rich ecosystem already existed, allowing us to conduct more interesting experiments and compare against a wider range of existing scenarios (e.g., the SDXL-DPO model).

As more specialized Flux variants become available, so too will the benefit of extending our approach to Flux-based flows. Until then, we suspect that the inclusion of such blocks will mostly serve to show that Flux is better than SDXL.

Developing a structured, intermediate representation

This is a great direction to investigate, but we believe there is enough to explore in this direction that it would warrant its own paper.

Marginal improvement over baselines (only 0.02 in GenEval)

A score difference of 0.02 on GenEval is not marginal. As can be seen from the results, DPO-tuned models outperform the baseline SDXL by 0.04 points. We improve over this by another 50%. Another point of reference: The difference between SD2.1 and SDXL, as reported in the GenEval paper, is 0.05 points, smaller than the gap between our approach and SDXL. We will provide this reference point in the text of our next submission.

Note that we also outperform all baselines on human preference metrics.

Concerns about incorrect number of objects in some generated images

We are confused by these remarks. Indeed, as with all methods, our approach sometimes generates an incorrect number of objects. The quantitative scores on GenEval, an established benchmark with hundreds of prompts, show that it happens less often for our method compared to the majority of baselines, and that overall, our approach outperforms all other baselines in prompt alignment.

Insufficient images in the paper

Our paper and supplementary include images generated by our method for 55 different prompts, in line with the standard in the field (e.g., SD3 has ~65, SDXL-DPO has ~45, SDXL has ~40, Pixart-Sigma has ~35, DALL-E3 has fewer than 20).

Comparisons to methods that replace the text-encoder with an LLM (Pixart-alpha, Lavi-Bridge, ELLA)

Flow #2 uses Pixart-Sigma (see line 313), which is a more recent, better version of the Pixart series. Our approach outperforms it even in a flow based setup.

Pixart-Alpha has been evaluated on GenEval in the SD3 paper, and has an overall score of 0.48. This is below even the baseline SDXL. Similarly, Lavi-Bridge and ELLA only have SD versions available, and we do not believe there is much insight to gain from adding comparisons against weaker, smaller-scale models.

Note that they also generally use more resources than our approach. ELLA report 14 days of 8*A100 GPUs, roughly double the compute budget of our data generation time + LLM tuning, after accounting for the fact we use an H100. Pixart-alpha use 753 A100 days.

I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.