GPT4LoRA: Optimizing LoRA Combination via MLLM Self-Reflection

1. The paper presentation needs significant improvement. Specifically, the top two figures of Figure 3 contain inconsistent information. Labels in the x-axis do not align with the legends. Legend in the bottom two figures of Figure 3 block the third bars. The authors should also be cautious of citation format, e.g., lines 305-306, line 215.
2. The results in Table 2 are not significant enough compared with LoRA-Composite in terms of CLIP scores. A significant test would be helpful.
3. Results in Table 3 show that without few-shot demonstrations, the performance is seriously downgraded, even inferior to LoRA merge. This makes me doubt the actual effectiveness brought by the "generate, feedback, and refine" pipeline.
4. Some details are missing for experiments. See questions below.

(1) The evaluation benchmark used in this paper is unclear. The paper mentions “Extensive experiments conducted on a benchmark of widely-used LoRA models” in lines 83 and 482, but lacks citations, leaving it unclear which text-to-image evaluation dataset is used.

(2) The paper lacks comparisons with comparable methods, such as ZipLoRA and LoRA Composite. ZipLoRA used DreamBooth and StyleDrop as evaluation datasets—could authors evaluate the GPT4LoRA on these datasets and choose ZipLoRA as the strong baseline model?

(3) The method's reliance on the self-reflection capabilities of multimodal large language models (MLLMs) like GPT-4 may result in variable outcomes depending on the MLLM's quality and adaptability, potentially limiting robustness across different models.

(4) While few-shot sample selection is critical to GPT4LoRA's success, details about this process are sparse, and the choice of demonstration samples significantly impacts performance, which may make it challenging for other researchers to reproduce the results effectively.

(5) A minor issue: some references in the paper are improperly formatted. For example, in line 144, "Reflexion Shinn et al. (2024) converts," and in line 214, "Unlike previous methods Lee et al. (2024); Xu et al. (2023)."

(6) Some results in the paper show limited improvements; it’s recommended that the authors conduct significance tests to analyze these improvements.

Major Issues:

1. Efficiency Concerns: GPT4LoRA’s efficiency is a significant concern. To combine multiple LoRAs into a single image, the proposed framework requires GPT-4o to first generate coefficients based on few-shot samples, then use SDXL to generate multiple images, followed by GPT-4o generating feedback to refine textual prompts and coefficients. Additionally, this process is iterated $N$ times. In sum, generating a single image can potentially require prompting GPT-4o over a dozen times and generating dozens of candidate images with SDXL. Given the relatively marginal improvements seen in qualitative and quantitative experiments, this substantial increase in computational cost may be difficult to justify.

2. Limited Experiments: Compared to LoRA Switch and LoRA Composite, GPT4LoRA's experiments are relatively limited. Specifically, this paper evaluates only 3-LoRA combinations with 105 composition sets, while previous works have explored 2-5 LoRA combinations and included more composition sets. Prior studies also incorporated human evaluation, which this work lacks. Analytical experiments are also sparse. For example, only an ablation on few-shot samples is provided, while there is no analysis of each step or the number of iterations in the framework. Since each step and additional iteration increases significant computational cost, analyzing these aspects would be valuable.

3. Inconsistent Claims: The authors' motivations appear unsupported due to inconsistencies. In the Introduction, the authors state that prior work suffers from (1) being "computationally costly and impractical when a large number of LoRA models are involved" (Lines 58-59). However, as discussed above, GPT4LoRA’s computation demands are notably higher than existing methods. They also claim (2) "A fundamental limitation of these methods lies in the subjectivity and unreliability of the evaluation process for image quality" (Lines 59-69), yet the MLLM self-reflection and GPT-based evaluation used here are established in prior work. Thus, the motivations behind GPT4LoRA seem unsubstantiated.

Minor Issues:

1. Overlap with Prior Work: Parts of this paper closely resemble the previous study [1]. For example, the entire "Diffusion Models" paragraph in 3.1, as well as the first two paragraphs in "LoRA Combination," show only minor paraphrasing of prior work. Besides, Table 1 is almost the same as Table 1 in [1] (but there is no mention of evaluation criteria and format requirements). In other words, the GPT-based evaluation approach (comparative evaluation, two evaluation dimensions, point-wise scoring, win rate, and evaluation prompts) is identical to [1] but lacks explicit citation and description.

2. Lack of Detail: Several key details are missing. See Questions 1-4 below.

3. Unexplained Results: Certain results seem confusing and lack analysis. See Questions 5-6 below.

Typos:

1. The legend and x-axis in the point-wise part of Figure 3 do not match.

References:

[1] Zhong et al. Multi-LoRA Composition for Image Generation.

• The paper is low on contributions - the method, while interesting, probably doesn't warrant a full paper - its better suited for a blog post.
• The paper is weak on quantitative results - Table 2 results does not appear statistically significant.
• The paper lacks analysis on the GPT4o outputs of the linear combinations.
• Experiments only conducted on closed source GPT4o, it is unclear if this kind of approach works for open source models, thereby limiting the applicability.