Unlocking Anticipatory Text Generation: A Constrained Approach for Faithful Decoding with Large Language Models

We would like to express our gratitude for your detailed review and feedback.

“The satisfaction score estimation is currently limited.”

In this work, we use a scoring mechanism to identify if a constraint has not been satisfied yet to guide the future generation for LLMs. Our work highlights the demonstrated effectiveness of the current constraint score function. Additionally, the appendix, particularly in Figure 5, presents supplementary results using the score function with binary Yes/No prompts. While acknowledging the potential exploration of other scoring functions, we posit that tuned models or the utilization of smaller models with more robust or accurate future satisfaction score functions may prove more beneficial.

“not clear if the gains will sustain for very long text generation.”

We present the model outputs of different decoding methods for the long-form QA task in Table 10. Notably, there is a reduction in error accumulation, with a significant decrease in the number of typos for the word '401k.' Additionally, the human evaluation results in Table 7 further underscore the improvements across three dimensions: Fluency, Informativeness, and Correctness.

“Automatic metrics is limited."

In our study, we surpass traditional n-gram-based evaluation metrics by integrating constrained satisfaction measures. These include the Coverage score for CommonGen, toxicity score from the Perspective API for the toxicity task, and correctness for the ALCE benchmark. Moreover, our research extends to human evaluations across three dimensions: Fluency, Informativeness, and Correctness. The combined outcomes from these diverse evaluations consistently emphasize the benefits of constraint satisfaction, ultimately leading to faithful generation.

“The factual correctness results are quite noisy”

Significantly, we observed that answers are truncated by the first newline in the ALCE's script, impacting our findings. Consequently, we present updated results on faithful question-answering generation. The new results indicate the superiority of our proposed method.

Thank you for your response, I will keep my score

We would like to express our gratitude for your detailed review and feedback. We have updated our submission with more details on experiments, decoding time, human evaluation results, etc.

“why |SEP| token is added?”

When calculating the future constraint satisfaction score R with the given prefix $y_{<=t}$​ , the use of <SEP> is employed to delineate the prefix from the language description constraint C. This separation is crucial; otherwise, the logistic score on the first token of C conditioned on the prefix $y_{<=t}$ lacks meaningful interpretation. Our preliminary experiments indicate that this setting is more effective. Further details will be expounded upon in the upcoming version.

“differences forms of work”

Indeed, there is other related work on guided generation, but the forms of control differ significantly. In our approach, we verbalize the constraint and estimate future satisfaction scores using LLMs. Our motivation stems from our belief in the expressive power of language. Several distinctions set our work apart from previous endeavors. Firstly, we do not require additional fine-tuning for tasks. Secondly, our guidance is integrated into the token-level generation process as opposed to post-generation refinement. Thirdly, we employ natural language constraints, providing flexibility to incorporate various constraints. Lastly, our approach includes self-interpretation of these partial prefix outputs, among other unique features.

“All the figure are not in the form of vector images.”

Thank you for bringing this to our attention. We have revised all figures, and we hope the updated versions meet your expectations. Your feedback is valuable, and if you have any further questions or need additional assistance, please don't hesitate to let us know.

“robust experimental evidence, one would expect more significant improvements.”

We have achieved notable results, particularly in the context of CommonGen, with substantial improvements in coverage scores (e.g., Falcon-7B-Instruct: 88.7% -> 93.3%, LLaMa-2-13B-Chat: 93.6% -> 95.2%, Falcon-40B-Instruct: 88.7% -> 97.6%). Furthermore, we provide updated results on QA faithful generation on LLaMa-2-13B-Chat, addressing the issue of answer truncation caused by the first newline in the ALCE’s script. In addition to quantitative outcomes, human evaluations confirm superior improvements across three dimensions: Fluency, Informativeness, and Correctness. We believe that future gains can be achieved with a more robust or accurate future-constrained score, potentially through the refinement or tuning of models.

“What is the tradeoff between time complexity and generated text quality?”

Our method incurs a linear slowdown of approximately $k$ times, primarily attributable to the overhead associated with computing future satisfaction scores on candidates. A detailed breakdown of decoding times for each example in our experiments is provided in Table 6.

It is worthwhile to note that the increase in decoding time is a reasonable trade-off for achieving faithful generation. To mitigate this, several strategies can be employed to maintain generation quality while reducing time, such as selecting a smaller $k$ and opting for smaller yet finely tuned LLMs capable of efficiently computing the future constraint satisfaction score $R(\y_{<=t}, C(\x))$.

We would like to express our gratitude for your detailed review and feedback.

“The novelty of the new constraint scoring function”

In this work, we use a scoring mechanism to identify if a constraint has not been satisfied yet to guide the future generation for LLMs. Our work highlights the demonstrated effectiveness of the current constraint score function. Additionally, the appendix, particularly in Figure 5, presents supplementary results using the score function with binary Yes/No prompts. While acknowledging the potential exploration of other scoring functions, we posit that tuned models or the utilization of smaller models with more robust or accurate future satisfaction score functions may prove more beneficial.

“Overall performance gains and only help small sized LLMs”

We have achieved notable results, particularly in the context of CommonGen, with substantial improvements in coverage scores (e.g., Falcon-7B-Instruct: 88.7% -> 93.3%, LLaMa-2-13B-Chat: 93.6% -> 95.2%, Falcon-40B-Instruct: 88.7% -> 97.6%). Furthermore, we provide updated results on QA faithful generation on LLaMa-2-13B-Chat, addressing the issue of answer truncation caused by the first newline in the ALCE’s script. In addition to quantitative outcomes, human evaluations confirm superior improvements across three dimensions: Fluency, Informativeness, and Correctness. We believe that future gains can be achieved with a more robust or accurate future-constrained score, potentially through the refinement or tuning of models.

“Does the <SEP> token seperate the prompt and continuation?”

No, the <SEP> token is specifically employed when calculating the future constraint score to separate the prefix and verbalized constraint. It's important to note that there is no <SEP> token used to separate the prompt and continuation.

“Does the inference mechanism ever lead to degenerate sequences? If yes, how often does that occur?”

We did not observe any degenerate sequences. The hyperparameters $k$ and $\lambda$ in Formula (2) can be adjusted to control the generation behavior.

Thank you for answering my questions.

We extend our sincere appreciation for your thorough review and constructive feedback. In response, we have enhanced our submission by including additional details on experiments, decoding time, human evaluation results, and other relevant aspects. Thank you for your valuable insights.

“Decoding time, tradeoff between time complexity and generated text quality ”

Our method incurs a linear slowdown of approximately $k$ times, primarily attributable to the overhead associated with computing future satisfaction scores on candidates. A detailed breakdown of decoding times for each example in our experiments is provided in Table 6.

It is worthwhile to note that the increase in decoding time is a reasonable trade-off for achieving faithful generation. To mitigate this, several strategies can be employed to maintain generation quality while reducing time, such as selecting a smaller $k$ and opting for smaller yet finely tuned LLMs capable of efficiently computing the future constraint satisfaction score $R(\y_{<=t}, C(\vx))$.

“Evaluation metrics”

We not only present evaluation metrics based on n-grams but also include constrained satisfaction scores, such as the Coverage score for CommonGen, toxicity score from Perspective API for the toxicity task, and correctness for the ALCE benchmark. Additionally, illustrative outputs are showcased in Table 9 and Table 10. Our study encompasses human evaluations across three dimensions: Fluency, Informativeness, and Correctness. Collectively, these findings highlight the benefits of constraint satisfaction, contributing to faithful generation.

“Lack experiments setup details”

We update the experiment setup and add more details on these models in Section A.1.