Stylus: Automatic Adapter Selection for Diffusion Models

We appreciate the thoughtful review and the constructive feedback! We hope the following clarifications address all the questions raised.

Quality Assurance for Adapters. We have taken careful preemptive measures to filter out problematic adapters and have outlined in Appendix A.7 a continual curation process, where we invite community members to report any adapters missed in our initial curation. Below, we highlight some key preventative measures for quality assurance and refer readers to Appendix A.7 and A.8 for a full discussion on safety and reliability of Stylus.

• Safety. Although LLM safety remains an actively evolving research area [1], StylusDocs employs a multi-stage filtering pipeline to identify and exclude problematic adapters. First, all explicit adapters tagged by Civit.ai are excluded from StylusDocs (Sec 3.2). Second, we use filters from Google’s VertexAI API to reject unsafe adapters based on LLM-generated descriptions. This especially catches problematic adapters that have innocuous original descriptions (Sec. A.7).
• Accuracy/Reliability. We note that, without access to the original fine-tuning dataset, we cannot guarantee the descriptions are completely error-free. However, we manually inspect commonly selected adapters in StylusDocs, blacklisting adapters that produce low quality images or cases where the StylusDocs description is inconsistent with observed adapter behavior (Sec A.4). Furthermore, authors are incentivized to oversell the model’s abilities. As such, Stylus’s refiner takes in example images in Civit.ai’s model card as a grounding mechanism to generate more accurate descriptions (Sec A.1).

Even with the safety measures we’ve taken, we acknowledge Stylus has the same risks of misuse as other public domain image generation tools including improper use for misinformation, producing explicit content, and reproducing copy-righted material from the training data. We emphasize the research prototype is not meant to be used in production without further application informed guardrails.

Refiner Ablation. In the following table containing CLIP and FID scores, we ablate Stylus's refiner:

The baselines are:

• SD v1.5 - The base Stable Diffusion model with the RealisticVision checkpoint.
• No-Refiner: Use base author-provided descriptions from Civit.ai or Huggingface.
• Gemini-Ultra Refiner: Use Gemini-Ultra as the Refiner’s VLM to generate better adapter descriptions. This is the version of Stylus presented throughout our paper.
• GPT-4o Refiner: Use GPT-4o as the Refiner’s VLM to generate better adapter descriptions.

These results show that a refiner is indeed important for textual alignment with the prompt. In fact, without a refiner VLM, Stylus performs poorly and chooses adapters that the composer thinks are aligned with the prompt but are not in practice. As a result, Stylus chooses adapters that hurt textual alignment (CLIP) and image quality (FID) and performs worse than base SD.

Furthermore, GPT-4o baseline performs much better than Gemini-Ultra, showing that better refiner descriptions can better aid the composer in selecting the right adapters. This also suggests that the Stylus’s performance is independent of Gemini-specific capabilities and benefits from further improved visual-language reasoning capabilities.

Choice of Gemini for Refiner. We chose the Gemini class of models since it has mature safety guardrailing. Specifically, Google’s VertexAI API provides stringent safety settings to block explicit content for the input prompt. Safety filters helped us filter out around 30% of original adapters that were tagged as non-explicit by Civit.ai.

Masking. The composer decomposes a prompt into tasks and assigns highly-aligned adapters per task. Next, a subset of candidate adapters are selected via masking for image generation. For each task, a mask either selects A) just one of the task’s adapters, B) all of the task’s adapters, C) or none of the adapters. To get the final selection of adapters, we randomly sample a mask for each task and merge the identified adapters to the original base model.

Regarding merging adapters, each adapter weights are first multiplied by the refiner’s recommended adapter weight, α_j (Eqn. 2). For example, Food Elegant Style LoRA recommends α=0.7 weight. Finally, to merge adapters into the base model, adapters weights are averaged per task and then summed across tasks.

We appreciate your feedback and believe these clarifications should address your concerns. We are open to further discussions to improve the paper. Thank you once again for your valuable insights!

[1] Hendrycks, Dan, et al. "Unsolved Problems in ML Safety." arXiv, 29 Sep. 2021, arXiv:2109.13916.

We appreciate the thoughtful and enthusiastic review!

We agree the incorporation of adapters/PEFTs [1] will continue to increase and that automatic adapter selection will be critical for managing and navigating the growing ecosystem of fine-tuned models. Stylus demonstrates improved diversity and visual quality evaluated quantitatively with automated metrics (CLIP/FID scores) as well as qualitatively via human evaluation and VLM as a judge.

Stylus provides a coherent strategy for composing and routing adapters for VLMs. Our retriever ablation (Sec. 4.3.1, Tab. 1) shows that naively selecting adapters (i.e. RAG) can lead to worse performance than the base SD checkpoint. Strategically composing and routing adapters unlocks a dimension of model performance previously underexplored.

We are excited to see LLMs used for dynamically selecting among models in related domains, including but not limited to:

• Automatic construction of agentic workflows/graphs. This includes using Stylus to decompose the task/prompt into a graph of subtasks and identifying which agent, among an ecosystem of agents, is best suited for each subtask.
• Routing between different base models from different providers to optimize the cost-performance tradeoff.
• Given a user prompt that requires composing multiple tools/functions, Stylus can identify, retrieve, and then compose the right sets of tools and functions for the LLM to invoke.
• Domain-specific fine-tuning is an emerging approach to reduce hallucination [2]. Stylus can select the right domain fine-tuned model to maximize factuality.

We are open to further discussions to improve the paper. Thank you once again for your valuable insights!

[1] Hugging Face Team. "Parameter-Efficient Fine-Tuning (PEFT) with Hugging Face." GitHub, 2023, https://github.com/huggingface/peft.

[2] Tian, Katherine, et al. "Fine-tuning Language Models for Factuality." arXiv, 2023, arxiv.org/abs/2311.08401.

We appreciate the thoughtful and enthusiastic review!

Reproducibility. We plan to release Stylus and StylusDocs as open-source resources to ensure transparency and reproducibility.

Adapter selection and masking. The composer decomposes the prompt into tasks and maps highly relevant adapters to each task. For more details on the Chain-of-Thought prompt [1] used for Stylus’s composer, refer to Table 2.

Next, a subset of candidate adapters are selected via masking for image generation. For each task, a mask either selects A) just one of the task’s adapters, B) all of the task’s adapters, C) or none of the adapters. To get the final selection of adapters, we randomly sample a mask for each task and merge the identified adapters with the original base model.

Regarding the merging step, each adapter weights are first multiplied by the refiner’s recommended adapter weight, α_j (Eqn. 2). For example, Food elegant style LoRA recommends α=0.7 weight. Finally, to merge adapters into the base model, adapters weights are averaged per task and then summed across tasks.

We appreciate your feedback and believe these clarifications should address your concerns. We are open to further discussions to improve the paper. Thank you once again for your valuable insights!

[1] Wei, Jason, et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” arXiv.Org, 28 Jan. 2022, https://arxiv.org/abs/2201.11903v6.

We appreciate the thoughtful review and the constructive feedback! We hope the following clarifications and experiments address your questions.

Efficiency. We note that Stylus's efficiency for image generation in terms of CPU and GPU resources is near identical to the base Stable Diffusion (SD) model. Aside from merging adapters (e.g. LoRAs) into the base model, which is small, the inference computation remains the same. Additional CPU memory is required to store unmerged adapters, as evidenced by efficient LoRA serving systems such as SLoRA [1] and dLoRA [2].

We emphasize that users today rely on manual search to identify a helpful subset of adapters. Stylus automates the process of determining the best set of adapters for a given user prompt, improving user efficiency. Furthermore, with recent releases of fast, high quality LLMs (i.e. GPT-4o-mini, Gemini 1.5 Flash), the composer’s latency will continue to reduce over time. We profiled GPT-4o-mini as the composer, which is over 3x faster than Gemini 1.5. This is a small overhead when compared to manual search over adapters.

Fairness. Stylus serves as an automatic enhancement to SD, leveraging additional training and data represented by LoRAs. Our retriever ablation (Sec. 4.3.1, Tab. 1) shows that naively selecting adapters (i.e. RAG) can lead to worse performance than the typical SD checkpoint. As such, we include base SD’s CLIP and FID scores as a reference for comparing different approaches to selecting and composing adapters.

Adapter descriptions. First, we note that, without access to the original fine-tuning dataset, we cannot guarantee the descriptions are completely error-free. However, we manually inspect commonly selected adapters in StylusDocs, blacklisting adapters that produce low quality images or cases where the StylusDocs description is inconsistent with observed adapter behavior (Sec A.4). Furthermore, we observe that over 80% of adapters on model platforms (Civit.ai/Huggingface) lack sufficient descriptions. As such, Stylus’s refiner takes in example images from Civit.ai’s model card as a grounding mechanism to generate more detailed and accurate descriptions (Sec A.1).

Furthermore, we add an additional refiner ablation that showcases that better adapter descriptions lead to performance gains. In the following table of CLIP/FID scores, we illustrate Stylus without and with refiner.

Here, the baselines are:

• SD v1.5 - The base Stable Diffusion model with the RealisticVision checkpoint.
• No-Refiner: Use base author-provided descriptions from Civit.ai or Huggingface.
• Gemini-Ultra Refiner: Use Gemini-Ultra as the Refiner’s VLM to generate better adapter descriptions. This is the version of Stylus presented throughout our paper.
• GPT-4o Refiner: Use GPT-4o as the Refiner’s VLM.

The quality of author-provided descriptions are poor, leading to worse performance than the typical SD checkpoint. Further improved refiner descriptions from GPT-4o can significantly boost Stylus’s performance, achieving the best textual alignment (CLIP) and image quality (FID) across all baselines, surpassing our original Stylus (with Gemini-Ultra).

Merging Adapters. We clarify Eqn. 2 averages adapter weights per task and sums adapter weights across tasks. We take several measures below to ensure that the second term in Eqn. 2 does not grow too large:

• Our masking scheme reduces the number of adapters in the final composition of LoRAs. (Sec 3.4)
• Empirically, with the COCO dataset, we observed the composer identifies at most seven tasks with associated adapters. We also have the option in the composer’s prompt to limit the number of tasks (Tab. 2).
• β scales down the rate at which the second term grows. We determined β=0.8 prevents highly-weighted adapters from overriding other concepts specified in the prompt, a challenge discussed in Fig. 13(b).

We appreciate your feedback and believe these clarifications should address your concerns. We are open to further discussions to improve the paper. Thank you once again for your valuable insights!

[1] Sheng, Ying, et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arXiv:2311.03285, arXiv, 5 June 2024. arXiv.org, https://doi.org/10.48550/arXiv.2311.03285.

[2] Wu, Bingyang, et al. "dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving." 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), USENIX Association, July 2024, pp. 911-927, Santa Clara, CA, www.usenix.org/conference/osdi24/presentation/wu-bingyang.