Visual Prompting with Iterative Refinement for Design Critique Generation

We thank Reviewer S1K2 for your insightful comments. We have revised the paper based on your feedback and uploaded the revised draft. We addressed your questions and concerns as follows:

Weakness 1: There is no qualitative results. Without good examples, I cannot concretely know what design critiques can be generated. For example, are they diverse enough? Can the difficult design critiques be generated? Without bad examples, I cannot clearly know the shortcomings of the proposed method.

We added Section A.4 in the Appendix that contains the results of a qualitative analysis. This section contains Figure 5, which presents two example outputs that demonstrate how our pipeline improves upon the baseline. Figure 6 (Appendix) includes two examples where our pipeline performed worse than the baseline. We have also updated Section 5.4 with a summary of these qualitative results. Please let us know if you want further clarification and examples.

Questions: How many rounds of iterative refinement are used in the experiment (averagely)? What will happen if we use less rounds of refinement or allow for more rounds of refinement?

Weakness 2: There is no discussion about the cost in terms of time or number of calls. The prompting pipelines involves many rounds of refinement and relies on the visual capabilities of LLMs. All of these will make it a slow one, which will influence the user experience.

We computed the average number of refinement steps for both bounding boxes (1.25 steps for Gemini and 0.88 for GPT-4o) and design comments (1.48 steps for Gemini and 1.17 for GPT-4o). If we reduce the rounds of bounding box refinement, the resulting IoU would likely approach that of BBox Generation (Table 2), and more comment + bounding box pairs may be removed during Validation. A similar effect is expected with fewer rounds of comment refinement.

We set a high limit of 10 refinement steps, and the iterative refinement rarely reaches this limit before deciding that the bounding box or comment is accurate. Therefore, allowing for additional rounds of refinement would likely not significantly affect the outcome. We added Section A.5 in the Appendix, which discusses and illustrates iterative refinement outputs, and also includes a brief cost analysis of the average number of refinement steps and the expected number of LLM calls for a full pipeline run.

Weakness 3: There is no comparison with methods based on fine-tuning, e.g., fine-tuning open-source multimodality LLMs like LLAVA or continually fine-tuning the one from Bravo et al. (2023). If fine-tuning based methods perform comparable or even better than the proposed prompting pipeline, we should use fine-tuning since the prompting pipeline with iterative refinement is slow and expensive.

Thanks for bringing up the general question of fine-tuning versus prompting. We want to emphasize that the focus of our paper is to investigate how far LLMs can go on critique tasks using prompting alone, which is a research question that has merits on its own. That said, we agree with you that fine-tuning might have advantages, especially when a more sizable dataset is available. Based on the UICrit dataset, we conducted additional experiments to address your question, and these details have been added in the revision. Namely, we have finetuned Llama 3.2 11b on UICrit for 3 epochs and have obtained the following results:

While finetuned Llama 3.2 has comparable Estimated IoU to our pipeline and higher comment similarity, qualitatively, it generates a very limited set of comments that cover text readability, visual clutter, and general comments on visual appeal. In contrast, our pipeline produces a more diverse range of critiques. Therefore, the fine-tuned model could not adequately replace the pipeline.

We have updated Table 2 with these quantitative results. We have added Figure 7 in the Appendix, which provides example outputs from both the fine-tuned Llama model and our pipeline to illustrate this, and we added the results of this qualitative analysis to Section A.4 of the Appendix. We have also updated Section 5.4 and the Discussion (Section 7) with this new finding.

Please let us know if you have any further questions or would like to see additional changes.

We thank Reviewer WLXH for your insightful comments. We have revised the paper based on your feedback and uploaded the revised draft. We addressed your questions and concerns as follows:

Weakness 1: There is limited technical innovation or scientific insight in the paper. Essentially, the authors report a way to couple together several LLM based processes to create a grounded description of the image in the context of UX design guidelines (derived from the classic Jakub HCI paper on the same). Whilst the design justifications for each stage are given, there is no real insight into the LLM’s capabilities or limitations. Why is an LLM able to do this? Which guidelines can it advise best on, and why? Rather the innovation is presented at face value, as an engineering result.

The cited paper by Duan et. al. (CHI 2024) has studied the capabilities and limitations of LLMs for design critique, justifying that LLMs have the potential to do this task based on their generalization capabilities to evaluate arbitrary UIs, reasoning abilities, and amenability to text-based design guidelines. They also found that LLMs advised the best on visual guidelines because visual guidelines contain more straightforward checks for aspects like color contrast and alignment. Since these topics have already been investigated by Duan et. al. (CHI 2024), we build upon these findings. For instance, our approach provides tailored fewshot examples to help general purpose LLMs better capture the nuances of design evaluation and address their limitations identified by Duan et al. (CHI 2024). We want to point out that our work is not just an engineering contribution. Our work contributes scientific insights in that it involves designing and investigating what works and what doesn’t by using LLMs as building blocks for a complex multimodal task. Additionally, we demonstrated the potential of this approach to be applicable and useful for other tasks.

This paper pushes the boundaries of what prompting alone can achieve for complex multimodal tasks. We provide insights into prompting for such tasks, specifically whether prompting can improve upon existing baselines for the complex multimodal task of design critique, how to implement it effectively, and whether the performance improvements from our method generalize to other multimodal tasks. For technical contributions, we introduce visual grounding with iterative refinement as a general technique applicable to various multimodal tasks and designed few-shot examples in a generalizable way that enhances performance. Our investigation focuses on the potential of LLMs using prompting alone, given that fine-tuning is highly resource-intensive in terms of data and computational requirements, which highlights the value of exploring lightweight prompting approaches. Without this work, there would be limited guidance on setting up these tasks with prompting alone to maximize performance.

Weakness 2: The paper introduces a formalised notation for the UI design critique tasks which seems unused later in the paper. Whilst it is welcome to have a clear task definition, I do not see the value of the math notation defined in Section 3 when it is not actually used later i.e. in the Method section of the paper. The use of this formalism appears to distract from what is otherwise largely an engineering based contribution sequencing multiple LLMs together.

Thank you for pointing out that the formalized notation for the UI design critique task is unnecessary. We also agree, and have removed the mathematical notation from Section 3.

Dear Reviewer WLXH,

Thank you very much for your responses. We are glad that our rebuttal and revision have addressed some of your concerns. Here we would like to further address your concerns regarding the experimental setup/rigor and the novelty. We sincerely hope you can reconsider your score given the progress we have made in addressing your points, and the additional information we provide here.

Re: the experimental setup and rigor

We understand your concern about the scale of the experiments. However, we hope you can evaluate the scale of our experiments in the context of multimodal critique generation, which is very complex due to its subjective and open-ended nature and requires participants with design expertise. Compared to previous studies on this topic reported in the literature, our study with human design expert ratings is significantly more comprehensive and rigorous than other studies involving human design experts. Unlike crowdworkers, human professional design experts are rare and are hence, challenging to recruit. Despite this, we conducted a study with 18 design experts who collectively evaluated 33 UIs, with each UI's comments rated by three designers to ensure validity. In comparison, other studies involving human design experts typically include only 5–7 experts [1, 2, 3, 4] or 14 experts [5], and evaluate considerably fewer UIs (6–17). Thus, we sincerely hope Reviewer WLXH takes into account these unique challenges we face in human evaluation for this task.

Re: novelty

Our work is the first to bring iterative refinement to the multimodal domain. There are many considerations and experimentations with trial-and-error to find the right design for the set of prompting techniques in our approach. Critique generation is a complex multimodal task, and our technique clearly outperformed the baselines. Our findings are valuable for other researchers to continue investigating the topic, and our approach has also shown promise in solving multimodal tasks beyond critique generation. We believe simplicity is the strength of our approach as it makes the work easily reproducible by others, and applicable to other problems, while still incorporating many details and careful designs in our approach. We sincerely hope you can share our view.

References:

[1] Peitong Duan, Chin-Yi Cheng, Gang Li, Bjoern Hartmann, and Yang Li. 2024. UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST '24). Association for Computing Machinery, New York, NY, USA, Article 46, 1–17. https://doi.org/10.1145/3654777.3676381

[2] Ziming Wu, Yulun Jiang, Yiding Liu, and Xiaojuan Ma. 2020. Predicting and Diagnosing User Engagement with Mobile UI Animation via a Data-Driven Approach. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376324

[3] Amanda Swearngin and Yang Li. 2019. Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 75, 1–11. https://doi.org/10.1145/3290605.3300305

[4] Camilo Fosco, Vincent Casser, Amish Kumar Bedi, Peter O'Donovan, Aaron Hertzmann, and Zoya Bylinskii. 2020. Predicting Visual Importance Across Graphic Design Types. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST '20). Association for Computing Machinery, New York, NY, USA, 249–260. https://doi.org/10.1145/3379337.3415825

[5] Eldon Schoop, Xin Zhou, Gang Li, Zhourong Chen, Bjoern Hartmann, and Yang Li. 2022. Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 36, 1–21. https://doi.org/10.1145/3491102.3517497

We thank Reviewer JBj1 for your insightful comments. We have revised the paper based on your feedback and uploaded the revised draft. We addressed your questions and concerns as follows:

Weakness 1: Minor Contributions: Compared to previous work, the authors primarily added the Validation and Iterative Refinement modules; however, these modules lack novel and crucial design. Additionally, the IOU performance of the box generation module only reaches around 35%, limiting its practical applicability.

Re: “novel and crucial design”, we believe our paper introduces several novel designs of prompting methods, which are carefully examined in a complex multimodal task. For instance, we developed a new method for fine-grained visual grounding by iteratively zooming into an image enhanced with coordinate markers, and generating few-shot examples by incrementally perturbing the bounding box to simulate the refinement process. We added more examples illustrating how the refinement progresses in the Appendix of the revision (e.g., Fig 8 and 9). Overall. we designed the prompts for each module and evaluated them extensively. Our paper pushes the boundaries of what prompting alone can achieve for a complex multimodal task (UI critique generation). The paper provides insights into prompting methods for such tasks, such as whether prompting can improve upon existing baselines, how to implement it effectively, and whether the performance improvements from our method generalize to other multimodal tasks. We believe these make valuable research contributions for others to explore multimodal prompting in the future.

Re: IOU performance, we want to clarify that, while the IoU of the box generation and refinement modules is around 35%, the IoU of the entire pipeline is actually 45.1%, according to the results from the human evaluation study (Table 3). This IoU improvement is due to the validation and text refinement modules.

Weakness 2: Poor Result Presentation: For both the design critique task and OVAD task, the paper provides minimal visual examples of the model's actual outputs. Instead, it predominantly uses ground truth images as illustrations, lacking clear demonstrations of the model's generated results.

Thanks for pointing it out. We have addressed the presentation issue by including more generated results in the revision. See our response for Question 1 below.

Question 1: Visualization of Results a) Showcase as many successful and failed cases as possible. Provide a clear analysis of improvements compared to previous models and discuss reasons for unsuccessful cases. b) Visualize the refinement process with intuitive displays of box and text feedback, as well as refinement results. This will help validate the roles of the respective modules.

We added Section A.4 in the Appendix that contains the results of a qualitative analysis. This section contains Figure 5, which presents two example outputs that demonstrate how our pipeline improves upon the baseline. These examples illustrate that our method reduces the occurrence of invalid, generic, and irrelevant design comments that are unhelpful, and produces more accurate, focused, and relevant feedback for the input screenshot and also produces more accurate bounding boxes. Figure 6 (Appendix) includes two examples where our pipeline performed worse than the baseline, where the pipeline eliminated a few valid design comments and generated a slightly less accurate bounding box. We have updated Section 5.4 with these additional qualitative results.

Additionally, we have included Figure 8 in the Appendix, which visualizes the iterative refinement process for the bounding box based on a design comment, and Figure 9, which illustrates the refinement process for a design comment based on a bounding box. Both figures show the final results of their respective iterative refinement processes.

(Continued From Above Comment)

Question 2: Trainable VLM Choice: Would directly fine-tuning advanced open-source VLM models, such as LLava or InternVL, on the UICrit dataset yield more accurate results?

With sufficient data, fine-tuning would eventually outperform prompting. However, prompting requires much less data and computing resources. In this paper, we meant to push the boundary to investigate how far prompting can go for this complex multimodal task. That said, we have conducted additional fine-tuning experiments to make the paper more complete. We finetuned Llama 3.2 11b on UICrit for 3 epochs and have obtained the following results:

While finetuned Llama 3.2 has comparable Estimated IoU to our pipeline and higher comment similarity, qualitatively, it generates a very limited set of comments that cover text readability, visual clutter, and general comments on visual appeal. In contrast, our pipeline produces a more diverse range of critiques. We have updated Table 2 with these quantitative results. We have added Figure 7 in the Appendix, which provides example outputs from both the fine-tuned Llama model and our pipeline to illustrate this, and we added the results of this qualitative analysis to Section A.4 of the Appendix. We have also updated Section 5.4 and the Discussion (Section 7) with this new finding.

Question 3: Simultaneous Generation of Box and Comment : Since comments correspond to specific boxes, humans typically focus on the area (box) first and then generate corresponding text. Generating the text before locating the box seems counterintuitive. How would generating the comment and box simultaneously impact the effectiveness of the model?

We agree that humans typically focus on the region (bounding box) before generating the corresponding critique. However, as we discuss in lines 203-207, we observed that LLMs often struggle with object grounding. If the model is asked to mark problematic regions first, it is more likely to produce inaccurate bounding boxes, leading to erroneous design comments based on those regions. This, in turn, increases the likelihood of errors in the subsequent refinement steps. We believe this finding will benefit others who will be designing prompts for multimodal tasks in the future.

Regarding simultaneous generation of both text and box, Duan et al. (CHI 2024) found that dividing major tasks for LLMs into more specific subtasks improves performance. Therefore, we separated the design critique and bounding box estimation into separate subtasks, following the method by Duan et al. (UIST 2024).

Question 4: Ambiguous Metrics : In both attribute detection and object detection tasks, mAP is a metric based on the percentage of successful samples, so performance differences are typically reported using absolute differences. However, in the paper, performance improvements are expressed as percentage increases in several sections, including the abstract, line 075, and line 483. This unconventional representation may introduce ambiguity.

Thank you for pointing out the confusing reporting of results. We have replaced the percent improvements with the absolute differences in the revision.

Please let us know if you have any further questions or would like to see additional changes.

Thank you for your answers. However, in my opinion, given the limitations in the paper's innovativeness, its performance in terms of applicability is more highly anticipated. As such, more comprehensive visualization results are expected. For example:

1. More usage examples are expected: The current effective visualization examples in the paper are limited to about 4-5, which lack persuasiveness.
2. Out-of-domain validation capability: The visualized UI interfaces in the paper seem somewhat outdated (e.g., uniform electronic buttons at the bottom, application-related information shown in the figures, etc., resembling design styles from 7-8 years ago). Since the paper utilizes advanced models like GPT, more modern UI interfaces are anticipated. In addition to, the ability to critique other systems' UIs and computer UI interfaces is also expected.

If the corresponding capabilities can be demonstrated, I would be more than willing to adjust my score upward.

As a follow-up to our previous comment, we tested the simple method that we discussed above, which utilizes the DOM tree to refine the generated bounding boxes through IoU comparison. This simple add-on to our approach noticeably improved the quality of the bounding box accuracy. We added Figures 14 and 15 (Appendix) in the latest revision to illustrate the new results and included Section A.4.3 in the Appendix to discuss this add-on and other potential refinement techniques that can enhance our method. This additional step could potentially be applied at the end of our pipeline to clean up the generated bounding boxes.

(Continued from Above Comment)

Weakness 3: The improvement upon (Duan et al., 2024b) in terms of comment quality (from 0.45 to 0.47 in Table 3) is small. For an incremental work, a more noticeable performance boost is expected, e.g., from 0.45 to 0.5 that almost lies midway between (Duan et al., 2024b) and Human.

While it is true that the performance boost for individual comment quality is less than expected, the comment set quality ranking is midway between (Duan et al., 2024b) and Human ((Duan et al., 2024b): 2.3, Pipeline: 2.0, Human: 1.7). The overall quality of the comment set is arguably more important than the average quality of each individual comment, as the feedback for a UI screen is typically presented as a set of comments. Also, participants were instructed to consider more aspects when ranking the comment sets, including the overall quality of the comments and comprehensiveness of the set.

Questions: In the human evaluation, why were the participants not asked to rate bounding box accuracy as in (Duan et al., 2024b)? Is “BBox IoU” in Table 3 computed by comparing with the ground truth bounding boxes? If so, is it possible for “BBox IoU” to penalize a predicted bounding box that is valid but different from any ground truth ones?

To ensure a rigorous and standardized metric for bounding box accuracy, we chose to compute the IoU, rather than rely on participant ratings, which would be more subjective and participants may have different scales on the degree of accuracy for the bounding box. The ground truth bounding box for each design comment generated by the baseline and pipeline were manually determined by the authors, who agreed on each ground truth bounding box.

Yes, it is possible for “BBox IoU” to sometimes penalize a predicted bounding box that is valid but different from the determined ground truth bounding box, so the BBox IoU would actually be an underestimate to the bounding box accuracy.

Please let us know if you have any further questions or would like to see additional changes.

We thank Reviewer PCXV for your insightful comments. We have revised the paper based on your feedback and uploaded the revised draft. We addressed your questions and concerns as follows:

Weakness 1: The scale of technical novelty is limited. What the paper actually does is to bring an existing idea, i.e., iterative output refinement for LMMs (Madaan et al., 2023; Xu et al., 2024a) into an existing problem domain, i.e., multimodal design critique generation (Duan et al., 2024b). The paper introduces several prompting techniques, such as including zoom-in image regions for the predicted bounding boxes into the visual prompts for bounding box refinement. However, the amount of novelty involved in these simple prompting methods is not significant enough. To increase the level of technical novelty, one possibility is to come up with more sophisticated prompting techniques, e.g., to generate better few-shot examples than those created via simple random perturbation for BoxRefine, Validation and TextRefine, or to iteratively modify the UI image beyond adding coordinate markers for more efficient bounding box refinement.

We acknowledge that the iterative refinement concept has been previously applied in various text-only domains (e.g., Madaan et al., 2023; Xu et al., 2024a). However, our work introduces novel adaptations of iterative refinement tailored to the multimodal design critique domain, which has unique challenges such as critique-specific visual grounding and refinement, and spatially contextual critique refinement. We also found that our method generalizes to other multimodal tasks beyond design critique.

Although our methods seem simple, which we feel it is a strength instead of a weakness; it took extensive investigation and iteration to design these methods. We considered other designs for prompting and fewshot examples, but found that our current setup had the best performance. For instance, we tried another few-shot design for bounding box refinement. This technique involves selecting the initial bounding box location based on visual similarity of the region it contained in the fewshot UI to that of the region contained by the input bounding box proposal of the input screenshot. This bounding box is then gradually moved closer to the ground truth bounding box for the fewshot UI to simulate the refinement process. However, we found that the simpler approach of randomly perturbing the bounding box actually gave better results (IoU 0.357 (random perturbation) vs 0.333 (visual similarity match)). We have updated Section A.2.1 in the Appendix with a description of this alternative few shot prompting method and a comparison of its performance with random perturbation (lines 702-708).

This paper pushes the boundaries of what prompting alone can achieve for complex multimodal tasks. We provide insights into prompting for such tasks, specifically whether prompting can improve upon existing baselines for the complex multimodal task of design critique, how to implement it effectively, and whether the performance improvements from our method generalize to other multimodal tasks. We feel these constitute valuable contributions for others to explore the topic further.

Weakness 2: The quantitative scores of (Duan et al., 2024b) in terms of Comment Similarity and Estimated IoU (used in Table 2) are missing, and should be added.

Thank you for pointing this out. We have updated Table 2 with the metrics from the baseline.

Dear Reviewer PCXV,

Thanks for your further comments. We are glad that our rebuttal has addressed some of your concerns. It seems that the only concern you have now is regarding the amount of technical novelty, which we want to further address here. Our work is the first to bring iterative refinement to the multimodal domain. There are many considerations and experimentations with trial-and-error to find the right design for the set of prompting techniques in our approach. Critique generation is a complex multimodal task, and our technique clearly outperformed the baselines. Our findings are valuable for other researchers to continue investigating the topic, and our approach has also shown promise in solving multimodal tasks beyond critique generation. We believe simplicity is the strength of our approach as it makes the work easily reproducible by others, and applicable to other problems, while still incorporating many details and careful designs in our approach. We sincerely hope you can share our view.

In the additional experiment and analysis that we conducted, which are already added to the revision, we found that our pipeline can also generalize to out-of-domain UIs, providing helpful design comments and bounding boxes to modern mobile UIs and websites using fewshot examples selected from only UICrit. Example outputs for these out-of-domain UIs are provided in Figures 11, 12, and 13 (Appendix). This finding further strengthens the practical utility of this pipeline.