DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation

Q3

The human annotation effort encompassed a comprehensive generated images of 18.9k samples (7 methods × 9 prompts × 150 images × 2 evaluations per sample). We employed 7 annotators from a labeling company, each evaluating approximately 2.7k samples over a two-week period with 8-hour workdays. Compared to ImagenHub's three-level scoring system [0,1,2], we expanded to a five-level scoring system [0,1,2,3,4], providing more granular assessment while achieving higher annotation consistency. To ensure robust evaluation quality, we implemented a systematic annotation protocol:

• Scoring Criteria Development: Paper authors (domain experts) developed detailed scoring standards based on generation results from 7 classic models. These standards were explicitly written into the GPT prompt, including specific Scoring Criteria and Scoring Range. We also selected typical examples (independent of the DreamBench++ dataset) as annotation references.
• Annotation Quality Control: We developed a dedicated annotation platform and implemented multiple rounds of pilot annotations. To ensure genuine annotation quality, these pilot annotations were not pre-disclosed to annotators. Early quality checks revealed typical issues like over-reliance on middle values (2 points), tendency to choose extreme values, and fluctuating annotation focus. Through continuous quality feedback and training, annotation quality reached expected levels after 3-4 iterations.
• Formal Annotation and Monitoring: During the formal annotation phase, we used a continuous quality monitoring mechanism. Quality assessment was conducted after each round, with decisions on additional training or proceeding to the next round based on results. This iterative feedback mechanism ensured overall annotation quality stability.
• Consistency Verification: The final human rating consistency significantly outperformed ImagenHub, validating the effectiveness of our annotation strategy. This achievement stems from a rigorous annotation process and annotators' professional commitment.

We invested substantial resources to ensure data quality, hoping this dataset will serve the broader research community, whether for model evaluation or training specialized evaluation models.

Q4

We also appreciate your suggestions, and we will evaluate more methods to support our conclusions in the final version.

Thank you very much for your thorough and insightful review of our paper. We greatly appreciate your time, effort, and the valuable suggestions you have provided, and we are committed to making the necessary revisions to enhance the quality of our work. Below, we address your comments in detail.

W1

Thank you for this important feedback regarding our choice of correlation metrics. We have conducted a thorough re-analysis of our methodology and results:

1. Metric Selection and Justification:

• We agree that Pearson correlation is more appropriate for comparing evaluation metrics with human ratings, as it effectively measures linear relationships across different rating methods and scales.
• While Spearman correlation was considered, it proved unsuitable due to our discrete 5-point scale ([0,1,2,3,4]), as rank-based analysis would be compromised by numerous tied rankings.
• We retained Krippendorff's Alpha specifically for human-human (H-H) and GPT-human (G-H) reliability measurements in our case, as we used identical evaluation criteria for both. This dual use (for H-H and G-H) enables direct comparisons with prior work like ImagenHub.

2. Updated Analysis: We have now computed Pearson correlations between all evaluation metrics and human ratings, providing a more rigorous quantitative comparison. This analysis will be added to the paper in final version:

3. Strengthened Claims: Our original conclusion that "Table 1 results show that DreamBench++ aligns better with humans than DINO or CLIP models..." is now supported by two complementary lines of evidence:

• The new correlation analysis quantitatively demonstrates stronger alignment between GPT and human evaluations
• This aligns with experience expectations, given that DreamBooth LoRA SDXL v1.0 is known to excel in Concept Preservation, unless some methods completely overfit the subject.

4. Data Transparency: To facilitate reproducibility, we have:

• Uploaded all rating data (GPT, DINO, CLIP, and human) to Google Drive
• Updated our anonymous GitHub repository (https://github.com/dreambench/dreambench_plus) with relevant analysis (Krippendorff's Alpha and Pearson Correlation) code
• Uploaded the original images generated by the method mentioned in the paper to Google Drive | Data | | --- | | DreamBench++ Evaluation Dataset | | Full Samples with 7 mehtods | | human rating data | | GPT rating data | | DINO rating data | | CLIP-I rating data | | CLIP-T rating data |

We appreciate your feedback, which has helped us strengthen both our analysis and presentation of results. The final version will include these updates to provide a more comprehensive and rigorous evaluation framework.

W2

Thank you for this insightful suggestion. We agree that a dedicated analysis of model performance across different generation types would strengthen our paper's alignment with its core objective of identifying superior technical approaches. We will incorporate the following comprehensive analysis in our revision:

• Concept Preservation Performance
  • Animal: Both fine-tuning and encoder-based personalization methods demonstrate strong performance in preserving animal concepts, primarily due to:
    • Consistent visual features within species from a human cognition perspective
    • For example, corgis exhibit highly unified visual representations, explaining their frequent use as test cases across personalization methods
  • Human: Encoder-based methods significantly outperform fine-tuning approaches in human concept preservation, attributable to:
    • Human observers' heightened sensitivity to facial feature accuracy
    • Encoder-based methods have more precise capture and injection of subtle facial features
    • Fine-tuning methods' tendency to focus on global image features rather than specific facial details
    • This also confirms that current Identity-Preserving Generation (like InstantID) prefer encoder-based methods
  • Object: The two approaches show complementary strengths
    • Fine-tuning methods:
      • Higher performance ceiling for common objects
      • Better handling of objects with standardized appearances
    • Encoder-based methods:
      • Provide reliable baseline performance for rare/complex objects
      • More consistent across diverse object types
  • Style: Both approaches perform well in style preservation, likely because:
    • Style features are more subjective and flexible
    • Style transfer requires capturing high-level visual characteristics
    • Both methods effectively learn and reproduce stylistic elements
• Prompt Following Capability
  • Base Model Impact: SDXL demonstrates superior prompt following compared to SD, driven by:
    • More sophisticated and bigger text encoder architecture.
    • Exposure to broader, more diverse training datasets and enhanced capacity for complex text-image relationships.
  • Methodological Differences:
    • Fine-tuning methods show clear advantages in prompt adherence due to:
      • Preservation of original text-image conditional distribution
      • Direct learning from image-text pairs
      • Better maintenance of semantic alignment when properly regularized
    • Encoder-based methods face challenges in prompt following due to:
      • Potential disruption of learned text-image relationships during feature injection

W3 & W4

Thank you very much for your suggestions on the presentation of the paper. We will include more detailed table explanations and more intuitive leaderboard tables in the final version.

Q1

Our choice of a 0-4 scale was deliberate and grounded in both empirical evidence and practical considerations. While normalizing to [0,1] would align with traditional metrics, our five-point scale (0,1,2,3,4) optimizes for human annotation reliability and score discriminability. Through empirical validation, we found that this granularity strikes an optimal balance - it provides sufficient resolution to capture meaningful differences while ensuring that each score point maintains a clear, distinct meaning to annotators. When we experimented with finer-grained scales, even experienced human annotators struggled to consistently differentiate between adjacent scores, resulting in lower inter-rater reliability as measured by Krippendorff's Alpha.

Q2

The validity of GPT's automated evaluations is substantiated through multiple verification mechanisms. First, we established strict scoring criteria for human raters, which were then used to guide GPT's evaluations (As GPT’s Prompt). To rigorously verify GPT's performance, we employed Krippendorff's Alpha to measure the agreement between GPT's scores and human ratings, treating GPT as an additional rater within our evaluation framework. The strong inter-rater agreement we observed indicates that GPT's reasoning and scoring closely align with human judgment patterns. Additionally, we manually examined the reasoning provided by GPT across different score levels to ensure that the explanations consistently reflect the assigned scores and adhere to our predefined criteria. This dual verification approach - quantitative through Krippendorff's Alpha and qualitative through reasoning analysis - provides robust evidence for the reliability of our automated evaluation system.

Thanks for the detailed response. I have adjusted the score accordingly (3 -> 6). Would love to see the continuous effort in maintaining the bench.

Q1: Yes, "comparing" was mistakenly written as "scoring". Thank you for pointing that out.

Q2: I'm a bit unclear about what the 'Why did the authors judge it as unsuitable for personalized generation and remove it?' means; can you explain it in more detail?

Q3: Regarding the example prompts shown in Figure 4: Yes, we generated 9 different prompts for each reference image using GPT for image editing (personalization). These 9 prompts vary in difficulty.

Q4: About the model scoring in Table 1: Table 1 does not clearly reflect the consistency between GPT and human evaluation; it should be combined with Table 4 for better understanding, and we will supplement the explanation in the final version. After discussions with reviewer RNsM, we realized that Krippendorff’s alpha is not a suitable metric for evaluating the consistency between DINO/CLIP and human evaluations. The reason is that the scoring criteria of DINO and CLIP differ significantly from those of humans, while Krippendorff’s alpha assumes consistent criteria across raters. To better measure consistency, the most appropriate metric is the Pearson correlation coefficient, which directly reflects the linear correlation between the model scores and human scores. Based on the above insights, we recalculated the relevant metrics using the Pearson correlation coefficient and updated the corresponding table in our paper. Despite this change, the main conclusions of our study remain consistent, further demonstrating the robustness and validity of our method.

We have now computed Pearson correlations between all evaluation metrics and human ratings, providing a more rigorous quantitative comparison. This analysis will be added to the paper in final version:

Q5: Regarding the claim in Figure 6: What we want to express is that many high-scoring cases selected by DINO maintain the same outline and posture (for example, the posture of the dog in the reference image and the generated image is the same, and both only show the upper body, but the facial details and colors are different; humans are more likely to think these are not the same dog, similar to the case of basketball players). Although the high-scoring cases from GPT may have many changes in posture/outlines and backgrounds, they exhibit stronger ID consistency.

Weakness 2 & 3: Possible hallucination of GPT-4o under subjective criteria

On the Use of Abstract Prompts with GPT-4o

We acknowledge the reviewer's observation that VLMs perform better with objective and specific queries (e.g., questions about object presence, location, or count). However, the personalized image generation task inherently involves subjective and abstract criteria. For example, determining whether a generated dog matches the reference dog goes beyond simple comparisons of size, color, or position. It often involves assessing nuanced and complex semantic relationships, such as whether the generated dog “is the same dog” as in the reference image. Similarly, textual instructions may contain a variety of intents, including altering specific attributes of objects while preserving others.

Our approach to GPT-4o scoring is justified primarily by its alignment with human evaluations, as evidenced by the high Krippendorff’s alpha between GPT-4o scores and human annotations. While GPT-4o operates as a “black box” to some extent, it provides justifications for its scores, which exhibit strong similarity to human reasoning. This consistency supports the feasibility of using GPT-4o as a proxy for human evaluation in this context.

On the Scoring System (0-4 Scale)

The choice of a 0-4 scale was carefully considered, balancing practical and empirical factors. While normalizing scores to [0,1] might align with conventional metrics, we found through empirical validation that a five-point scale (0, 1, 2, 3, 4) offers better granularity for capturing meaningful differences in image quality. This scale ensures clear and distinct score meanings for annotators, optimizing annotation reliability and discriminability.

When testing finer-grained scales, we observed a significant drop in inter-rater reliability, as even experienced annotators struggled to consistently differentiate between adjacent scores. The five-point scale mitigated these issues, resulting in higher Krippendorff’s alpha values and improved consistency.

On the Number of Human Annotators and Annotation Protocol

The human annotation process was extensive and rigorous, covering 18.9k generated samples (7 methods × 9 prompts × 150 images × 2 evaluations per image). Seven professional annotators evaluated approximately 2.7k samples each over a two-week period with 8-hour workdays.

To address concerns about annotation consistency and quality, we implemented a robust protocol:

1. Scoring Criteria Development: The paper authors, as domain experts, developed explicit and detailed scoring standards based on results from seven classic models. These standards were incorporated into the GPT prompt, including specific scoring criteria and example-based references to ensure clarity.
2. Quality Control: Multiple rounds of pilot annotations were conducted to identify and address common issues, such as over-reliance on middle scores, preference for extremes, and inconsistent focus. Iterative feedback and training improved annotator performance to the desired level within 3-4 iterations.
3. Formal Annotation: Continuous monitoring during formal annotations ensured quality stability. Each round was followed by quality assessments, with decisions made regarding further training or progression based on performance metrics.
4. Consistency Verification: Final human rating consistency significantly surpassed benchmarks like ImagenHub, validating the effectiveness of our annotation process.

Our resource-intensive efforts underscore our commitment to creating a high-quality dataset that can serve the research community for both model evaluation and as a resource for training evaluation-specific models.

We acknowledge that a larger number of annotators could provide additional robustness. However, we emphasize that our protocol and iterative training ensured that the final annotation quality and consistency exceeded existing benchmarks. This focused approach allowed us to maintain a balance between resource constraints and dataset quality, ensuring that every annotator adhered to rigorous standards.

We thank you for your acknowledgement of our work and appreciate your suggestions. We are committed to answering your concerns in detail and making the necessary revisions to enhance the quality of our work.

Weakness 1: Comparison with newer metrics such as TIFA

Thank you for your suggestion and reference. The need of comparing our work with existing text-to-image (T2I) benchmarks has also been raised by other reviewers. We will include a more comprehensive experiment that compares our method with existing T2I benchmarks in the appendix of the final version. Here, we would like to highlight some of the key dinstinctions between our benchmark and other existing benchmarks.

• Designed for Different Tasks. Existing benchmarks such as TIFA are mostly designed to evaluate text-to-image (T2I) generation tasks, i.e., generating an image based on text description. In contrast, our proposed DreamBench++ focuses on the personalized generation task, a more complex challenge that requires generating an image based on not only text instructions but also a reference image (e.g., given a reference picture of my dog, generate a picture of my dog on the moon).
• Different Criteria. The core criterion of T2I benchmarks is fine-grained text adherence, i.e., assessing whether the generated image accurately reflects detailed textual instructions. Additionally, our DreamBench++ not only focuses on text adherence but also visual concept preservation (e.g., assessing how well the dog in the generated image preserves the unique characteristics of the dog depicted in the reference picture).
• Different Methodology**.** TIFA evaluates generated images by posing a series of VQA-based questions. For instance, given the text description "A person sitting on a horse in air over gate in grass with people and trees in background," TIFA might ask questions such as “What is the animal?”, “Is there a gate?”, or “Is the horse in the air?”. The answers are then used to compute a fine-grained score.

Predictably:

• In terms of prompt following ability (text adherence), our method may not perform as well as benchmarks specifically designed for T2I tasks. This is because:
  • Our design does not emphasize fine-grained text adherence;
  • Instead, we use a scoring mechanism similar to that for concept preservation, where GPT determines scores based on provided criteria and ranges.
• Nevertheless, even with this relatively simple design, the correlation between GPT’s scores and human evaluations consistently exceeds 90% (noting that we will update Table 4 in response to Reviewer RNsM's feedback by replacing Krippendorff’s alpha with Pearson Correlation Coefficient, with the results remaining unchanged).

Dear Reviewer Xmsj,

Thank you for your great reviewing efforts and your supportive assessment of our work! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply.

We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any.

Best,
Authors

Thank you very much for your thorough and insightful review of our paper. We greatly appreciate your time, effort, and the valuable suggestions you have provided, and we are committed to making the necessary revisions to enhance the quality of our work.

Below, we clarify our contributions as well as the distinction between our work and existing benchmarks in response to your concerns.

1. Our Motivation of Creating DreamBench++

Existing T2I benchmarks (e.g., HRS-Bench and DALL-EVAL) focus on the text-to-image generation task, the core criteria of which is fine-grained text adherence, i.e. whether the generated image correctly reflects the detailed requirements in text instructions (e.g., generating objects in specific quantities or generate compositional objects). Some benchmarks also includes evaluation on image quality, and potential biases (e.g., regarding race and gender). For instance, HRS-Bench highlights strict adherence to textual instructions, such as generating images involving specific quantities or logical compositions (e.g., a cat chasing a dog), and evaluates models across multiple dimensions, including image quality, fairness, and social bias.

In contrast, the core motivation of DreamBench++ is to assess the personalized generation task, which is a more challenging task that generates an image based on not only text instruction but also a reference image (e.g. generate a picture of my dog on the moon). In addition to the criteria mentioned above, the evaluation of personalized generation tasks also involves measuring the similarity between the subject in the generated image and the subject in the reference image. This task is distinct from traditional text-to-image evaluation methods and aims to address the lack of effective evaluation for personalized generation capabilities in existing benchmarks.

2. Limitations of Existing Evaluation Methods

We notice that while existing evaluation methods are good at evaluating image-text alignment, they do not perform well in assessing the subject similarity between images, as exemplified in Figure 6. The most intuitive and accurate evaluation approach would be to manually assess the qualities of generated personalized images, i.e. conducting user studies. However, current user studies suffer from notable limitations:

• Lack of unified evaluation standards: Different papers often adopt inconsistent evaluation metrics and processes, making the results difficult to compare.
• Lack of transparency: Specific evaluation details are rarely disclosed, making it challenging to reproduce or verify conclusions objectively.

As a widely adopted proxy to human evaluation, traditional automated metrics (e.g., DINO Score and CLIP Score) are insufficient in reflecting the subject-level similarity between the generated and reference images. As shown in Table 4, these metrics exhibit low correlation with human subjective evaluations. (Note that we would have updated Table 4 following Reviewer RNsM's feedback to replace Krippendorff's alpha metric with Pearson Correlation Coefficient, the outcome stays the same)

3. Technical Innovation

To address these complex challenges, we propose a universal and robust evaluation framework for personalized generation, leveraging multimodal large models to quantify subjective similarity. The core innovations of this approach include:

• Shifting from comparison to scoring tasks: Initially, we attempted a comparison task where the multimodal model judged which of two generated images was closer to the reference image. However, experiments revealed severe positional bias—the model tended to favor the first image. To mitigate this, we transitioned to a scoring-based method, designing detailed prompts to guide the evaluation process. This shift significantly improved the stability and accuracy of the evaluation.
• Validating the effectiveness of the evaluation framework: We compared scores generated by the multimodal model with human-assigned scores and demonstrated the framework’s effectiveness and robustness using Pearson correlation. This method provides a unified, reproducible benchmark for personalized evaluation, addressing a key gap in current research.

Reviewer RNsM also discusses this point in his proposed weakness2. We plan to add another section to reinforce this point as well. Dedicated analysis of model performance across different generation types would strengthen our paper's alignment with this core objective of identifying superior technical approaches. We will incorporate the following comprehensive analysis in our revision:

• Concept Preservation Performance
  • Animal: Both fine-tuning and encoder-based personalization methods demonstrate strong performance in preserving animal concepts, primarily due to:
    • Consistent visual features within species from a human cognition perspective
    • For example, corgis exhibit highly unified visual representations, explaining their frequent use as test cases across personalization methods
  • Human: Encoder-based methods significantly outperform fine-tuning approaches in human concept preservation, attributable to:
    • Human observers' heightened sensitivity to facial feature accuracy
    • Encoder-based methods have more precise capture and injection of subtle facial features
    • Fine-tuning methods' tendency to focus on global image features rather than specific facial details
    • This also confirms that current Identity-Preserving Generation (like InstantID) prefer encoder-based methods
  • Object: The two approaches show complementary strengths
    • Fine-tuning methods:
      • Higher performance ceiling for common objects
      • Better handling of objects with standardized appearances
    • Encoder-based methods:
      • Provide reliable baseline performance for rare/complex objects
      • More consistent across diverse object types
  • Style: Both approaches perform well in style preservation, likely because:
    • Style features are more subjective and flexible
    • Style transfer requires capturing high-level visual characteristics
    • Both methods effectively learn and reproduce stylistic elements
• Prompt Following Capability
  • Base Model Impact: SDXL demonstrates superior prompt following compared to SD, driven by:
    • More sophisticated and bigger text encoder architecture.
    • Exposure to broader, more diverse training datasets and enhanced capacity for complex text-image relationships.
  • Methodological Differences:
    • Fine-tuning methods show clear advantages in prompt adherence due to:
      • Preservation of original text-image conditional distribution
      • Direct learning from image-text pairs
      • Better maintenance of semantic alignment when properly regularized
    • Encoder-based methods face challenges in prompt following due to:
      • Potential disruption of learned text-image relationships during feature injection

W4

Thank you for pointing this out. In fact, this was one of the core motivations behind constructing our benchmark. Through our exploration of personalized generation, we identified significant limitations in the current mainstream metrics (e.g., DINO and CLIP), particularly their inability to accurately distinguish between overfitting and true generative quality when evaluating models’ personalization capabilities.

Specifically, DINO and CLIP metrics often place excessive emphasis on the visual similarity between the generated and reference images. Many works achieve high scores by overfitting to the reference image, where the generated subject is nearly identical to the reference. However, such results fail to reflect true personalization capabilities. In contrast, DreamBooth, when appropriately fine-tuned, can generate images that align well with the text description while retaining the subject's identity. Although this approach may lead to lower DINO/CLIP scores, it demonstrates better personalization, as the subject in the generated images exhibits meaningful variations in actions, poses, expressions, and appearances.

To address this issue, we devised a more interpretable and discriminative evaluation method by combining two metrics: Concept Preservation (CP) and Prompt Following (PF). By calculating the product CP·PF, we can better quantify the overall performance of a model in personalized generation while avoiding biases from single metrics. Additionally, we observed that the CP/PF ratio can provide insights into overfitting: higher ratios often indicate cases where the generated images excessively mimic the reference image at the expense of faithfully following the text description.

In the final version of the paper, we plan to include a more detailed table and explanation in the appendix to clearly present the performance of different models. Below is a summary of our experimental results:

Q

Thank you for your careful review and insightful questions. Below are our detailed responses:

1. On the calculation and rationale for using Krippendorff’s alpha We calculated Krippendorff’s alpha using the implementation provided by the open-source repository fast-krippendorff. We chose this metric because it is widely regarded as the most flexible and robust measure for evaluating inter-rater reliability. It effectively handles missing data and supports various data types, including ordinal and continuous data. Furthermore, Krippendorff’s alpha has been widely adopted in related studies, such as GPT4Eval3D and ImageHub. This makes our choice both theoretically sound and conducive to comparisons with results from these works.
2. On the origin of the numbers 54.1% and 50.7% The numbers 54.1% and 50.7% were calculated as follows: 1) 54.1%=79.64%−25.54%; 2)50.7%=93.18%−42.48%. Here, 79.64% and 93.18% represent the agreement scores between GPT and human evaluations, while 25.54% and 42.48% represent the agreement scores between DINO and human evaluations. These differences quantify the relative improvement in alignment of GPT over DINO when compared to human evaluations.
3. On the appropriateness of using Krippendorff’s alpha for DINO/CLIP consistency with humans After discussions with reviewer RNsM, we realized that Krippendorff’s alpha is not a suitable metric for evaluating the consistency between DINO/CLIP and human evaluations. The reason is that the scoring criteria of DINO and CLIP differ significantly from those of humans, while Krippendorff’s alpha assumes consistent criteria across raters. To better measure consistency, the most appropriate metric is the Pearson correlation coefficient, which directly reflects the linear correlation between the model scores and human scores. Based on the above insights, we recalculated the relevant metrics using the Pearson correlation coefficient and updated the corresponding table in our paper. Despite this change, the main conclusions of our study remain consistent, further demonstrating the robustness and validity of our method.

W3

Regarding the concern about our benchmark using only a single image per instance, we appreciate the reviewers' perspective and would like to clarify our rationale for this design choice.

• Practical considerations: In many practical applications, especially in scenarios where personalization is required, the availability of multiple reference images for each instance is often limited. Therefore, we designed our benchmark to assess the capability of personalization using a single image, which we believe reflects real-world constraints.
• We primarily focus on dataset diversity: While we highly appreciate Benchmarks that contains multiple images like DreamBooth and CustomConept101, our work adopts a slightly different focus. Our primary emphasis is on ensuring the diversity of the dataset in terms of styles and types of objects. We believe this approach facilitates a more comprehensive and balanced comparison across various personalized image generation methods.
• Performance are sufficient with single images: We have compared generation qualities of generation results on DreamBench by using multiple images and single image per instance, and we found that tested methods such as SDXL can achieve their full capabilities even with just a single image. Therefore we believe that our benchmark is sufficient in evaluating the full capabilities of these methods within the scope of our research.
• Difficulty in constructing datasets of intance with multiple images: Collecting multiple diverse images for the same instance is challenging, and in our existing personalization dataset, the diversity of instances with multiple images is not high. We lacked an effective way to address this issue at the time, and ignored the excellent CustomConcept101 dataset at the time, so we did not consider constructing a dataset of instance with multiple images. In the final version, we will expand our dataset based on CustomConcept101 to improve the diversity, fairness, and robustness of our evaluation framework.
• We leave this for future research: We fully concur with the reviewers that collecting multiple diverse images for a single instance is indeed a valuable approach. While we respectfully maintain that utilizing a single image per instance does not diminish the significance of our benchmark or the validity of our main results, we deeply appreciate the reviewers' suggestion in this regard and recognize this as an excellent direction for future research. We will discuss the potential for expanding our dataset in this manner in the final version of the paper.

W2

Thank you for raising this valuable question. We agree that validating the transferability of our method is an important scientific concern. To address this, we conducted additional tests on various multimodal foundation models and provided comparative results in our experiments.

In the last column of the table, the ratio (i.e., the average proportion of H-H consistency) quantifies the alignment between model evaluations and human evaluations, reflecting overall performance. The experimental results indicate the following:

1. Performance of the GPT-4o series: The capabilities of GPT-4o are continuously evolving. We observed that the 2024-08-06 version achieved higher human-machine consistency compared to the 2024-05-13 version. Furthermore, the mini version (GPT-4o-mini-2024-07-18), despite its smaller size, also achieved notable results.
2. Performance of Claude 3.5: We also tested Claude-3.5-sonnet-20241022, which is considered to have capabilities comparable to GPT-4o. This model exhibited a similar level of consistency ratio, further supporting the transferability of our method.
3. Performance of Gemini: Due to API availability limitations, we could only test an earlier version of Gemini-1.5-pro-001, which performed relatively weaker. The newer versions of the model may demonstrate improved consistency.

Overall, our results indicate that the proposed method is not only applicable to GPT-4o but also generalizes well to other advanced multimodal models. This cross-model consistency enhances the practical usability and scientific value of the method. Thank you for helping us refine this argument through your insightful question.

Thank you very much for your thorough and insightful review of our paper. We greatly appreciate your time, effort, and the valuable suggestions you have provided, and we are committed to making the necessary revisions to enhance the quality of our work. Below, we address your comments in detail.

W1

Thank you for the your insightful comments regarding the dataset’s sources and diversity. We address these concerns as follows:

First, regarding the data sources, our dataset is not strictly limited to three websites. Google Images is a comprehensive search engine that accesses images across the entire web. The primary reason for selecting these three sources lies in their copyright transparency and suitability for academic use. Ensuring proper copyright compliance is critical for constructing a dataset, as it avoids potential issues that have led to the removal of some datasets. Furthermore, these sources provide high-quality images with sufficient diversity to meet our needs.

Second, regarding dataset diversity, our goal was to maximize diversity with a relatively small number of images, considering the cost-intensive nature of GPT-based evaluations. To achieve this, we designed a rigorous construction pipeline (illustrated in Fig. 4) that ensures diversity at multiple stages:

1. Keyword Generation: We started by categorizing personalized tasks into broad groups (e.g., animals, humans, styles, and objects) and used LLMs to generate a large number of keywords. Additionally, we leveraged the top 200 keywords from the Unsplash dataset and included numerous human-proposed keywords. This step ensures semantic diversity in the dataset’s sources.
2. Data Retrieval: Using Google Images, we retrieved images from across the web, inherently ensuring diversity due to the wide range of available content.
3. Manual Filtering: After data retrieval, we conducted manual filtering to remove images with highly repetitive styles or categories, further enhancing the dataset’s diversity and quality.

While our diversity evaluation primarily relies on t-SNE visualization, the results clearly demonstrate broader data coverage and reduced clustering. Furthermore, every step in our pipeline is explicitly designed to prioritize diversity. We acknowledge that there is room for improvement in quantifying diversity, and we plan to explore more comprehensive evaluation methods in future work.

In conclusion, despite the limited sources and simple evaluation methods, we believe our dataset demonstrates significant advantages in diversity and robustness, supporting future personalized tasks effectively.

Dear Liang,

Thank you for your comment! We appreciate your recognition of our work and focus on the challenging task of personalized image generation. We are committed to continuously pushing the scale and diversity of the benchmark.

Best,
Authors