COLLIE: Systematic Construction of Constrained Text Generation Tasks

The format of the instructions (i.e., constraint types) are identical across different sources, e.g., "Generate a passage with $n$ sentences."

However, the distribution of constraint values (e.g., $n$ in the previous constraint) differ for different sources. For instance, Wiki has twice the average number of sentences compared to CC-News (See Fig. 3(c)).

Let us know if you have further questions, thanks!

Thanks for your great suggestion! We have incorporated the related work and discussion in the draft!

Thanks for your positive comments! Let us know if you have further feedback.

1. Some important details for instruction rendering should be moved to the main paper.

   • As per your suggestion, we have moved more details for instruction rendering to the main paper (section 3, in blue).

2. The paper mentions that the technique can be used for constraining words, word blacklisting, however a qualitative analysis is missing for the same in the current version.

   • By word blacklisting we mean something similar to constraint para02: "Generate a paragraph with at least 4 sentences, but do not use the words “the”, “and” or “of”." We have complete model outputs in our supplementary material, but we can also add some of these examples to the appendix. Thank you for the suggestion.

3. Section 5: Performance enhancement through feedback - It might help to list down the details of how the feedback is being used.

   • Thanks! We have moved some details for use of feedback to the main paper, as per your suggestion (section 5.1, in blue).

Thanks for your feedback! Let us know if your have followup questions.

1. The idea is not very novel, and similar ideas have already been explored. For example, [1] also constructed a dataset using similar constraints for instruction fine-tuning.

   • Thanks for bringing up this concurrent work, which we have cited and discussed in the revised paper!
   • It focuses on turning constraints into NL instructions and fine-tuning smaller language models such as T5-11B, and the constraints themselves are fairly simple and limited (only 5 types). In contrast, Collie focuses on making constrained text generation more flexible, customizable, and harder, challenging even SoTA LLMs like GPT-4.
   • We believe these two efforts are orthogonal and can be complementary --- we can use these harder and more flexible constraints from Collie-v1 and other Collie constructed datasets to diagnose and analyze LMs (as shown in the paper), and improve the instruction tuning of LMs.

2. The dataset may not be very useful. Specifically, because the rules are too vague/arbitrary, the extracted ground truth is not useful for the evaluation process: the authors only use them for comparing fluency. In addition, since the rules can be arbitrarily designed, this compiled dataset does not hold much value, because many similar datasets can be compiled with different engineering details.

   • The extracted ground truth text is not only useful for comparing fluency, but also to ensure the extracted constraint values are solvable using natural text. We believe it is actually important to evaluate constraint satisfaction using the constraints instead of the extracted text, as there could be very diverse generations that all satisfy the same constraints.
   • As stated in the paper, our main contribution is twofold: the Collie framework for constructing constrained text generation tasks, and the Collie-v1 dataset using a concrete and preliminary set of constraint types. The facts that “rules can be arbitrarily designed” and “many similar datasets can be compiled with different engineering details” are exactly the merit of our proposed Collie framework --- while previous datasets feature a fixed set of simple constraints and ad-hoc pipelines for data collection, the Collie framework allows automatic and scalable construction of arbitrary grammar-expressible constraints.

3. In the limitations, the authors mentioned potential problems with filtering and processing functions. Could the authors elaborate on what issues might exist for the extraction process?

   • Every text corpus has artifacts (e.g., tables, footnotes, references, headings that might not be natural sentences or paragraphs). We design filtering and processing functions to patch or eliminate passages with these artifacts. However, it is impossible to design a filtering system that is perfect, so there could be downstream reference texts or constraints that look unnatural. This is more of a fundamental limitation with any approach that leverages text in the wild, and not a limitation specific to Collie or CTG construction. We have revised the limitation part to better explain this, thanks!