We thank the reviewer for their positive review and useful feedback! We address the weaknesses below:

The presentation, particularly in the experimental section, could benefit from clarification and better organization to enhance readability and comprehension.

We have uploaded a new draft of the paper which we hope greatly improves the readability and structure. In particular, Section 4.2 shows changes in blue text with improvements to clarity in the experimental setup, and a correction to a section reference in the baseline description of Random-Search

The paper occasionally employs exaggerated language and makes promising claims that seem to lack sufficient empirical backing. For example, the paper states “providing a recipe that seemingly generalizes to many domains and modalities” and “it is often easier for a model to evaluate the quality of a generation than to generate the same high-quality text.” These claims would be more convincing if supported by concrete evidence or reference or discussion.

We take this concern seriously and take a moment to clarify on the two highlighted points to resolve the misunderstanding. On the quote “providing a recipe that seemingly generalizes to many domains and modalities”, we highlight in the conclusion through reference that AI feedback can be done with any model that has been instruction-tuned such as the AI feedback model used in QDAIF. We cited in the conclusion Liu et al., 2023, a work that introduces instruction tuning for multimodal text-image understanding models. Instruction tuning enables this model to solve tasks in image and text domains, including solution evaluation - it is interesting future work for example to apply AI feedback to images with the visual instruction-tuned model (or another similar image-text model such as GPT-4+V) to evolve images and creative art, that can be described by AI feedback on the new domain of images, and generate refinements of images.

On the quote “it is often easier for a model to evaluate the quality of a generation than to generate the same high-quality text”, we cited Saunders et al., 2022, to refer to the generation-discrimination gap described by the quote. We also cited this part of their work in Section 4.4 in the original manuscript (now in A.18), but realized that we forgot to include page number 12 in the first mention of this reference (which we did in Section 4.4 originally), containing the reference to generation-discrimination gap which gives context to our results. The page number is added now to the first mention of the reference. We apologize for the lack of clarity, and hope the added detail will prevent future readers from sharing the same confusion.

A separate conclusion section, summarizing the key findings and contributions, would be advantageous for providing clear takeaways for the readers.

Thank you for sharing this suggestion and for thinking of ways to improve the presentation of QDAIF. We added to the first paragraph of the conclusion a brief statement on key takeaways from results for our draft revision, to support the communication of the contributions messaging in the final paragraph of the conclusion.

Conclusion:

We would be grateful for your thoughts on the quality of our manuscript and contributions of QDAIF following our response here as well as the general comment, and hope to get your support in realizing QDAIF as a valuable contribution to ICLR, during the remaining time of the discussion period.

If you feel that your comments have been adequately addressed, we would greatly appreciate it if you could update your score to reflect that.

Thank you for your encouraging statements on the strengths of the paper. We look forward to hearing more from you on ways to make this paper stronger.

We understand that the author discussion period is closing soon, but we are still open to coming back to you on any remaining questions you may have about the manuscript. We have gone deep into following up on your concerns, and have resolved them from our understanding of your points raised.

We would be happy to quickly answer any potential remaining questions from you. Thank you again for your work on improving the paper, and encouraging statements on the strengths of our work.

We thank the reviewer for their thoughtful review of the paper. We come back to the reviewer’s questions and concerns below, starting off with some clarifications to aid in the understanding of our findings and explanations, and then in the second part of the reply clarifying on main concerns.

The evaluation result seems to be a bit confusing; for example, in Table 1, one of the methods is LMX, Fitness-only, then in section 4.3 when it explains the method, there is LMX Quality-only. Is that the same method as LMX, fitness-only

Thank you for highlighting this for correction! It was a typo on our part, “LMX, fitness-only” should be worded as “LMX, Quality-Only”. We corrected this in Table 1 in the revised draft, shown in blue text. The Quality-Diversity (QD) literature frequently uses the term “fitness” for the notion of quality as a metric, for general context.

QD metric is used throughout, which is the “sum of highest quality value found in each bin” - it seems to only focus on quality rather than diversity. For readers not familiar with QD metrics, some explanation/justification of why QD measures Quality and Diversity will be helpful

In many domains you want to get a wide range of high-quality diverse artifacts (the motivating case as described in the introduction); the QD score [1] is one popular measure used in the literature. QD score increases are possible in two ways - by filling a previously empty bin with a solution of that kind (i.e. finding a novel solution, with improvement to the overall archive by the amount of quality improvement introduced from the addition of this new solution of a particular quality value), or by improving over an existing solution in a filled, non-empty bin with an evolved solution that is evaluated to be of this kind/bin, but has a higher quality score, as described in the method overview in Section 3 regarding the acceptance of evaluations to the archive. Coverage of the archive with new, diverse solutions, is, therefore, a key contributor to improvements in QD score (which we highlight in new additional stats including coverage in the added A.7 section, referenced in figure 3).

Table 1: there is a lack of explanation of the quality metrics. For example, what is the difference between human QD score and quality rating? Is quality rating from humans?

Yes, quality rating is from humans. Thank you for sharing your concern on readability here. We provide full details on the human evaluation study referenced in the Table 1 results in Appendix A.1 and reference it in the “Evaluation” paragraph of Section 4.1. We added minor clarification and further self-contained referencing of A.1 in the revision for improved clarity.

On page 5, section quantifying performance and diversity, it is unclear where the probability comes from in the sentence “ the solutions’ quality estimate is derived from the logarithm of the probability of the LM’s answer” … Please clarify

As part of the standard AI feedback method, the quality and diversity prediction values are obtained from LM next-token-prediction log probabilities, where we assess the probability of the AI feedback LM in predicting for a given text/story one label/token(s) (e.g. “horror”) versus another label/token(s) (e.g. “romance”). We quote in the section referred here in the paper, “The log probability of these responses serves as our measure of solution diversity” for context, applying the same approach to quality AI feedback. We slightly reworded a part of this sentence, highlighted in blue text. Thank for raising possible suggestions to improve the readability of the paper.

We have added additional material in the v2 revision to further expand upon the topic of Automating QDAIF Search (Diversity Axes).

We hope that this gives you further insights into the strengths of QDAIF. We have followed up on all of your concerns, and see that we have resolved them from our understanding of your points.

We would be happy to quickly answer any potential remaining questions from you. Thank you again for your work on improving the paper.

Could we have a detailed comparison with other AI feedback methods or discuss how QDAIF specifically addresses their limitations? Yes, we highlight the relevance of AI feedback (AIF) methods in the context of QDAIF in the original manuscript in Section 2.3, and added minor clarifications in this section in the revised draft, providing more clarity on how QDAIF leverages AIF in a new setting to introduce a new kind of QD algorithm, enabling other recent advancements in QD research studied in different domains to be applied on top of the high-level QDAIF approach.

We show with this clarification that QDAIF is a complementary direction of research that takes inspiration from works applying AIF such as Constitutional AI [1] and Self-Refine [2], and their proven results in utilizing AI feedback for qualitative text quality assessment (which led to LM refinements being able to improve text solutions), to apply AIF to a new context towards tackling QD problems (adding the diversity and evolution aspects for a new application in QD).

We hope that this brings a clearer picture of the differences between QDAIF and other AI feedback methods, where QDAIF enables QD approaches in new domains through qualitative AI feedback.

Can you provide more insight into the scalability and computational efficiency of QDAIF in more complex and large-scale tasks?

The scalability and computational efficiency of QDAIF both hinge on the capabilities and sizes of available existing LMs (which carry out both the mutation of texts, and the evaluation of text solutions defined by AI feedback prompts).

We refer to an additional analysis carried out and described since the original manuscript version to study the effect of the generator/mutator LM’s model size in Section 4.3, on “LMX Model Size”. Results summarize likely gains in scalability, where discovered samples of creative writing solutions found by QDAIF (with larger model sizes as generator LMs) obtained higher human feedback scores for quality compared to their ratings of samples from smaller LMs.

Furthermore, our additional analysis on the qualitative sample poems in the Poetry domain in the revised manuscript, Appendix A.18 (in particular, Figures 21 and 22) suggests potential improvements in meaningful mutations of texts towards more diverse, higher-quality texts when running QDAIF with larger, more capable models such as GPT-4 compared to GPT-3.5 Turbo. While QDAIF benefits from more search iterations (meaning more LM generation and evaluation calls via label/attribute prediction), it still shows QD score performance sample efficiency improvements over existing baselines, hinting at the potential scalability of QDAIF that can improve with advancements to available LMs.

Conclusion:

We would be grateful for your thoughts on the quality of our manuscript and contributions of QDAIF following our response here as well as the general comment, and hope to get your support in realizing QDAIF as a valuable contribution to ICLR, during the remaining time of the discussion period.

If you feel that your comments have been adequately addressed, we would greatly appreciate it if you could update your score to reflect that.

Thank you again for your work on the review.

[1] Constitutional AI - https://arxiv.org/abs/2212.08073

[2] Self-Refine - https://arxiv.org/abs/2303.17651

We thank the reviewer for their insightful review and thoughtful comments on the strengths of the paper. We come back to the reviewer’s questions and concerns below.

QDAIF still requires researchers to define the axes of diversity they are most interested in, which may limit its autonomy in creative search.

Thank you for raising this interesting discussion point following the presented statements on the limitations of QD methods in the final section of the manuscript. We added Appendix A.10 which investigates and tackles the concern raised here through additional experiments and literature review, and referenced A.10 in the conclusion section on the limitations discussion.

Firstly, to give more context from prior works in the literature (as we described in the context of the discussion on limitations since the original manuscript), most existing algorithms since the first introduction of QD search have not yet studied approaches (if at all) to alleviate the need to define the axes of diversity, especially ones that would likely be most interesting to users in a creative search. In A.10, we added references for an existing possible direction of defining higher dimensions of diversity via unsupervised diversity representation learning. Here we noted limitations of this approach in the case where for our desired use cases, we do want to leverage the general knowledge of assessing all types of creative writing from foundation models to inspire creative QDAIF search; we can then exploit LM domain knowledge for expanding on possible output text solutions where the distinctions in diverse, high-quality texts can be subjectively appreciated and informatively cataloged.

We describe in A.10 an approach where such domain knowledge for automating the initialization or expansion of diversity axes can be explored for more diverse, high-quality texts. With recent advances in LM capabilities (e.g. GPT-4), we can specify the search problem or domain of interest, and ask it, for example, to directly generate ideas for plausible new diversity axes to explore, or even directly generate the AI feedback prompts which may be used in the existing QDAIF setups, maybe following the structure and format of say the feedback prompts we tested.

The question remains - how effectively would QD score performance (the measure of quality and diversity in archive solutions) improve if one tested such a pipeline, and continued search with the introduction of additional diversity axes over time.

We demonstrated in A.10 results that applying dimension expansion on the Stories domain as an example is effective in enabling the improvement of QD score in higher dimensions of diversity, surprisingly even providing gains in “Best Solution Quality” (mentioned in general response).

Overall, new results that we found and added bring us closer to the potential of automatically defining diversity axes, proving practical performance gains to be had.

The generalizability of QDAIF to other domains and tasks beyond creative writing is not extensively discussed… How does the proposed QDAIF approach perform in other domains and tasks beyond creative writing?

We added Appendix A.20, referenced in Section 4.4, to show the results of QDAIF applied to coding problems, which is not focused on creative writing, and compared to the Random-Code baseline.

We are encouraged that you highlighted the importance of applying QDAIF to highly-subjective, qualitative domains, where no viable measures except for AI feedback (the alternative to expensive human feedback) exist, unlike the domains studied by QD works in prior work. Interestingly, we show in new results that AI feedback’s ability to bring evaluation to qualitative domains is also effective for code-writing tasks, where QDAIF generated qualitatively more diverse sorting algorithms (from first principles) with higher quality scores obtained, and overall higher QD score than that of the baseline (that returned nearly always one particular type of sorting algorithm, 95% of the time).

We appreciate the interest of the reviewer in QDAIF when applied to domains beyond creative writing. We further highlight the generalizability of QDAIF with this finding.

We have added additional material in the v2 revision to further expand upon the topic of Automating QDAIF Search (Diversity Axes) and an Additional Domain Beyond Creative Writing.

We hope that this gives you further insights into the strengths of QDAIF. We have followed up on all of your concerns, and see that we have resolved them from our understanding of your points.

We would be happy to quickly answer any potential remaining questions from you. Thank you again for your work on improving the paper.

We thank the reviewer for their insightful review, and are encouraged by their comments on the strengths of the paper. We come back to the reviewer’s concerns on experimental soundness below.

In terms of baseline, they seem to be simple variations of the framework. It would be needed to have baselines cited in prior work. The results are significantly better for QDAIF, but it is unclear whether this is due because the baselines are bad or whether the model is really better… My only concern is the lack of more sophisticated baselines.

Thank you for sharing your feedback on the importance of detailed and informative baseline comparisons. We agree with this statement, and revised the draft with additional baseline comparisons, introducing new baselines from prior works (on top of the LMX, Quality-Only baseline introduced in the original manuscript, and studied in the LMX, or Language Model Crossover paper [1]) and detailing performance comparison results with QDAIF in Appendix A.8, with detailed implementation details of added diversity-seeking baselines in A.9. In addition to further demonstrations of performance improvements by QDAIF compared to baselines, the new comparisons highlight the importance of each component of QDAIF in improving the QD score performance during search, with new insights on the value of diversity-seeking elements of relevant search algorithms, often outperforming the baseline which optimizes for quality in solutions only (LMX, Quality-Only).

Furthermore, as highlighted in the general response, we introduced a new variant of QDAIF for Poetry, LMX-rewrite (in edited Section 4.4), which becomes a fair comparison to the Random-Poems baseline, where both methods do not request for solutions of specific genres and tones in poetry, but LMX-rewrite adds the rewriting mutation and evolution of archive bin solutions to improve over Random-Poems. This experiment improvement also enabled us to conduct an ablation with the method Fixed Seed Rewrite, which reveals that the element of evolving an improving population with QDAIF (in addition to rewriting mutation) is important for significant performance improvements with QDAIF.

Given the improvements to the experimental soundness with sophisticated baselines, and a showcasing of the strengths and weaknesses of baselines, we have shown performance gains from QDAIF to be meaningful (with improved insights on the contributions of components of QDAIF), in line with seminal works in the literature which also compare to baselines (such as Random-Search, as well as LMX, Quality-Only for single objective quality/fitness optimization), in the experiments of the Novelty Search [2] and MAP-Elites [3] papers.

I would ask the authors to evaluate their proposed approach on another task, such as control text generation (e.g., writing about specific topics and using a classifier to assess the topic being discussed)

Thank you for your interest in the applicability of QDAIF to other domains in addition to creative writing such as the existing Opinions, Stories, and Poetry domains.

Firstly, the Opinions domain seems to be relevant to the idea of topic writing, which is highlighted in the review here in the paper explanation. The topic of eating vegetables and plant-based foods would likely be relevant here.

Still, we take the suggestion on the value of studying QDAIF in different kinds of domains into account, and added results of QDAIF applied to a new (non-creative-writing) code-writing domain in Appendix A.20. We saw marked improvement in the diversity and quality of generated code from QDAIF (LMX-rewrite) for the task of implementing sorting algorithms to solve a specified problem, compared to the Random-Code baseline.

We believe that results here also demonstrate the flexibility and generality of QDAIF, in being helpful to creative search in another, more-technical domain.

Conclusion

We would be grateful for your thoughts on the quality of our manuscript and contributions of QDAIF following our response here as well as the general comment, and hope to get your support in realizing QDAIF as a valuable contribution to ICLR, during the remaining time of the discussion period.

If you feel that your comments have been adequately addressed, we would greatly appreciate it if you could update your score to reflect that.

Thank you again for your helpful feedback that further improved the paper’s quality!

[1] Language Model Crossover (LMX) - https://arxiv.org/abs/2302.12170

[2] Novelty Search - https://stars.library.ucf.edu/facultybib2010/1530/

[3] MAP-Elites - https://arxiv.org/abs/1504.04909

We have added additional material in the v2 revision to further expand upon the topic of Additional Domain Beyond Creative Writing and an Additional Baselines.

We hope that this gives you further insights into the strengths of QDAIF. We have followed up on all of your concerns, and see that we have resolved them from our understanding of your points.

We would be happy to quickly answer any potential remaining questions from you. Thank you again for your work on improving the paper.