Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leaps

Because the field of jokes is quite new, we appreciate your feedback so that we can better express our innovation.

1. Not well explained (like Problem definition).

Humor reasoning is a multi-hop process, and each hop is based on external knowledge injection and proper rationales. Without them, it is difficult for the model to understand the internal humorous logic, making it prone to pattern recognition. Therefore, AAIE is proposed to inject and augment knowledge into the original training data. It will be facilitative for LLMs to understand the underlying logic and rationale. And then in the stage of GESIT, the reasoning logic for each online generated response is extracted using GPT-4o. In this process, external knowledge is introduced again to assist the model in logic reasoning and human preferences learning. Experimental results demonstrate that the combination of these two methods can both enhance the model's judgment ability and improve its generative capability.

For example, there is a conversation, “What do you think of Aeroflot?” “I was on this airline once, and arrived an hour early, and finally I have been divorced three years.”. Aeroflot has nothing to do with divorce seemingly. But they could be related by this way, "Aeroflot $\rightarrow$ Well-known feature is fast $\rightarrow$ It could arrive an hour early $\rightarrow$ It leads to trip disrupted $\rightarrow$ So something to ruin the marriage was found $\rightarrow$ Finally the responder is divorced." From the example, we can conclude that there is a sparse knowledge graph between question and answer. The key of making a creative leap is to mitigate the information insufficiency issue. To realize it, the injection of knowledge beyond the literal is necessary to reasoning, which is the goal of CLoST.

2. Datasets details.

The in-house data is private and is not suitable for disclosure. But We have collected Ruozhiba dataset and generate responses by CLoST. They are manually reviewed and screened to be public as research tool. And then we will publish the full code after the paper is accepted.

3. The evaluation metrics and experimental setup.

With respect to the multiple-choices questions, there are varying difficulty levels in the dataset, and it is variation primarily reflected in the choice options. We collect data from varied humor generation games such as Origi-GO. In these games, a question gains many responses from human and these responses are ranking by human voting. We select response with different votes numbers to construct choices. Easy case is formed by responses with large disparity. Hard case is formed by small difference votes' responses. And with develop of options' number, the gap between options will shrink. Finally, we clarify that we randomly shuffle the options in training and validate sets.

4. Experiments.

We appreciate your concerns about fairness, but as an admittedly best LLM, GPT-4o is inevitably used as a baseline for comparison. To allay your concerns, we supplemented an experiment on the Ruozhiba dataset, which most well-known LLMs have been trained on. We asked GPT-4o to rewrite the Ruozhiba query into a question and answer pair, placing the punchline in the answer section. Then, we asked GPT-4o again to rewrite the ground-truth answer into a non-humorous version. Based on the positive-negative pair data, LLMs were tested, and the results are shown in Table 9 (we modify it in line 1034 of revised paper). The results show that CLoST also realizes state-of-the-art performance on Ruozhiba dataset.

We conduct a human evaluation to validate CLoST's performance in humor generation (\textcolor{blue}{line 430}). We choose the first 200 samples in the validation split of the Ruozhiba dataset\footnote{https://github.com/Leymore/ruozhiba/tree/main?tab=readme-ov-file} and the method mentioned above is to make the query into questin-answer pair. Then four LLMs generate responses to each question as four options. Then, we conduct a user preference study to directly verify the creativity of the LLMs. We present question and several corresponding replies, and ask users to choose the most creative and humorous responses. We select four advanced LLMs to generate responses for a total of 200 questions, and the four responses from the four distinct LLMs are randomly permuted in the options. We conduct an extensive survey through an online survey platform\footnote{https://www.wjx.cn/}, ultimately collecting 15 valid questionnaires with 3000 votes. Within these collected questionnaires, we calculate the proportion of times each LLM is selected for each question. Finally, we aggregate the total number of times each LLM is chosen across all validation samples, as shown in Figure 6(c). The ratio of this sum to the overall number of selections among all LLMs signifies the user preference for each LLM. Also we calculate the win rate based on the dimension of the problem as shown in Figure 6(b).

Training pipeline (we modify it in line 783 of revised paper): CLoST takes a two-stage training strategy. In the first process (supervised fine-tuning (SFT)), we randomly initialize a LoRA model. And we train the model with single-turn question-answer format data (data from Figure 2(a)(b)(c)) and muti-turn question-answer format data (data from Figure 2(d) and AAIE). In the second process (Direct Preference Learning (DPO)), the first stage model serves as ref model, and it is frozen as judgement model. The tunable model is trained to improve the reasoning generation capability. At the beginning of stage 2, only preference question-answer data without rationale is fed into the tunable model for training. After several steps, the rationale for each online generated response is extracted using GPT-4o and the preference question-answer data with rationles are mixed into the original dataset. And in each batch, the ratio of 'w' and 'w/o' rationale is $1 : 1$.

Experiments details (we modify it in line 792 of revised paper).: Our model is fine-tuned based on QWEN1.5-32B-Chat with fine-tuning method LoRA on 8 A100 GPUs. For the first stage, we train the model on the $95\%$ of dataset mentioned above for 6 epochs with AdamW optimizer and the learning rate of $3e-4$. In the second stage, $5\%$ of the dataset are used to train GESIT for 3 epochs with AdamW optimizer and the learning rate of $2e-4$. The models are tested in the tasks introduced in previous part. And the parameters used in generation are list in Table 5.

This article solves the problem that the answer from most LLMs is not humorous and creative enough. In the future, in order to make it more widely used, we will consider the HG problem and offensive or culturally insensitive content you raised.

Dear Reviewer SKMr,

I hope this message find you well. We have carefully considered your feedback and have made significant improvements to the manuscript. We truly value your insights, and your expertise has greatly contributed to enhancing the quality of our work. Could you please let us know if the revisions meet your expectations? As the deadline for discussion nears, we kindly ask if you could review our updated paper. We are eager to address any additional questions that may arise. Thank you for your invaluable support and consideration.

Sincerely,

Authors

4. Experiments.

(a) Regarding the experiments on generation, we conduct Divergent Association Task to show the model's imagination capability in Figure 5 (we modify it in line 411 of revised paper). In this experiment, we provided specific words and asked the model to generate associations and imaginations, resulting in 10 associated words. We then used these 11 words to calculate the DAT score where semantic distances are computed. The results show that CLoST gains the best performance on divergent association thinking. For humor generation, there are some showcases in Appendix (we modify it in line 912 of revised paper). All showcases show that CLoST could be more human-like and short which leaves more room for imagination.

(b)We conduct a human evaluation to validate CLoST's performance in humor generation (we modify it in line 430 of revised paper). We choose the first 200 samples in the validation split of the Ruozhiba dataset\footnote{https://github.com/Leymore/ruozhiba/tree/main?tab=readme-ov-file} and the method mentioned above is to make the query into questin-answer pair. Then four LLMs generate responses to each question as four options. Then, we conduct a user preference study to directly verify the creativity of the LLMs. We present question and several corresponding replies, and ask users to choose the most creative and humorous responses. We select four advanced LLMs to generate responses for a total of 200 questions, and the four responses from the four distinct LLMs are randomly permuted in the options. We conduct an extensive survey through an online survey platform\footnote{https://www.wjx.cn/}, ultimately collecting 15 valid questionnaires with 3000 votes. Within these collected questionnaires, we calculate the proportion of times each LLM is selected for each question. Finally, we aggregate the total number of times each LLM is chosen across all validation samples, as shown in Figure 6(c). The ratio of this sum to the overall number of selections among all LLMs signifies the user preference for each LLM. Also we calculate the win rate based on the dimension of the problem as shown in Figure 6(b).

(c) We appreciate your concerns about fairness, but as an admittedly best LLM, GPT-4o is inevitably used as a baseline for comparison. To allay your concerns, we supplemented an experiment on the Ruozhiba dataset, which most well-known LLMs have been trained on. We asked GPT-4o to rewrite the Ruozhiba query into a question and answer pair, placing the punchline in the answer section. Then, we asked GPT-4o again to rewrite the ground-truth answer into a non-humorous version. Based on the positive-negative pair data, LLMs were tested, and the results are shown in Table 9. The results show that CLoST also realizes state-of-the-art performance on Ruozhiba dataset. (we modify it in line 1034 of revised paper).

(d) We conduct ablation study about judgement performance in Table 3 and 4 (we modify it in line 379and 447 of revised paper). In Table 3, row 5-6 shows that model trained with AAIE realizes a noticeable improvement on English benchmarks. Row 3-4 in Table 4 also shows the increasing of performance on Chinese benchmark. And ablation study on Divergent Association Thinking test is conducted in Figure 5(b) (we modify it in line 411 of revised paper). which shows that AAIE enhance the divergent associate thinking ability.

Because the field of jokes is quite new, we appreciate your feedback so that we can better express our innovation.

1. The motivation is not clear.

Humor reasoning is a multi-hop process, and each hop is based on external knowledge injection and proper rationales. Without them, it is difficult for the model to understand the internal humorous logic, making it prone to pattern recognition. Therefore, AAIE is proposed to inject and augment knowledge into the original training data. It will be facilitative for LLMs to understand the underlying logic and rationale. And then in the stage of GESIT, the reasoning logic for each online generated response is extracted using GPT-4o. In this process, external knowledge is introduced again to assist the model in logic reasoning and human preferences learning. Experimental results demonstrate that the combination of these two methods can both enhance the model's judgment ability and improve its generative capability.

For example, there is a conversation, “What do you think of Aeroflot?” “I was on this airline once, and arrived an hour early, and finally I have been divorced three years.”. Aeroflot has nothing to do with divorce seemingly. But they could be related by this way, "Aeroflot $\rightarrow$ Well-known feature is fast $\rightarrow$ It could arrive an hour early $\rightarrow$ It leads to trip disrupted $\rightarrow$ So something to ruin the marriage was found $\rightarrow$ Finally the responder is divorced." From the example, we can conclude that there is a sparse knowledge graph between question and answer. The key of making a creative leap is to mitigate the information insufficiency issue. To realize it, the injection of knowledge beyond the literal is necessary to reasoning, which is the goal of CLoST.

2. Judgement model's importance.

Judgement ability is a fundamental skill of LLMs to further empower their reasoning ability. Some researches have verified that reward model is important for reasoning methods such as Monte Carlo tree search or the reinforcement strategy learning. Reward model helps to optimize the behavior of large language models by providing feedback so that they produce outputs that are more in line with expectations. Due to the subjectiveness of humor, a scalar score may include great noise. Choosing-best-from-multiple-choices is then a fundamental to establish an evaluation metric.

3. CLoT is not be claimed clearly, and humor generation performance compare.

CLoT develops two basic ability to facilitate humor understanding and generation, selection skill and ranking skill. The innovation of CLoT is the introduction of nouns as instruction receipt, thereby enhancing generalization. However, as mentioned in CLoT, it directly fine-tuning on the given creative data merely amounts to a rigorous fitting of the data. This fitting process only captures the inherent creative patterns within the data, failing to stimulate ”thinking outside the box” for generating novel ideas. Furthermore, creative data is inherently scarce, and relying solely on dataset fitting easily leads to being trapped in local patterns. (we modify it in line 49 of revised paper).

Regarding the humor generation, humor is subjective, and judgements from diversified groups may vary. In terms of expression differences, CLoST could be more human-like and short which leaves more room for imagination. From a statistical perspective in Figure 6(b), human evaluation shows that CLoST gain more preference.

Dear Reviewer gQyc,

I hope this message find you well. We have carefully considered your feedback and have made significant improvements to the manuscript. We truly value your insights, and your expertise has greatly contributed to enhancing the quality of our work. Could you please let us know if the revisions meet your expectations?  As the deadline for discussion nears, we kindly ask if you could review our updated paper. We are eager to address any additional questions that may arise. Thank you for your invaluable support and consideration.

Sincerely,

Authors

Because the field of jokes is quite new, we appreciate your feedback so that we can better express our innovation.

1. AAIE's detail like motivation, examples, output, stability .

Humor reasoning is a multi-hop process, and each hop is based on external knowledge injection and proper rationales. Without them, it is difficult for the model to understand the internal humorous logic, making it prone to pattern recognition. Therefore, AAIE is proposed to inject and augment knowledge into the original training data. It will be facilitative for LLMs to understand the underlying logic and rationale. And then in the stage of GESIT, the reasoning logic for each online generated response is extracted using GPT-4o. In this process, external knowledge is introduced again to assist the model in logic reasoning and human preferences learning. Experimental results demonstrate that the combination of these two methods can both enhance the model's judgment ability and improve its generative capability.

For example, there is a conversation, “What do you think of Aeroflot?” “I was on this airline once, and arrived an hour early, and finally I have been divorced three years.”. Aeroflot has nothing to do with divorce seemingly. But they could be related by this way, "Aeroflot $\rightarrow$ Well-known feature is fast $\rightarrow$ It could arrive an hour early $\rightarrow$ It leads to trip disrupted $\rightarrow$ So something to ruin the marriage was found $\rightarrow$ Finally the responder is divorced." From the example, we can conclude that there is a sparse knowledge graph between question and answer. The key of making a creative leap is to mitigate the information insufficiency issue. To realize it, the injection of knowledge beyond the literal is necessary to reasoning, which is the goal of CLoST.

The difficulty in AAIE process can be concluded in two points: 1. How to deepen and broaden understanding step by step. 2. How to end the thought process. For example in Table 10-15, a simple prompt is difficult to stimulate the understanding of the model like row 1. AAIE utilize LLMs' world knowledge to gradually make the generated prompt contain more information as shown in Table 12 row 1 to 3, simulating the process of human gradual in-depth thinking. So the answers from the two prompts are different like Table 13. Additionally, imaginator tries to revivification the conversation based on the informative prompt's answer as shown in Table 10 row 1-2. It makes a shift on story line. So it guides the entire system to explore the boundaries of the conversation, which makes the understanding broader. Finally, If the punchline or core idea of imagined conversations is completely deviated from the original, the system will stop injecting new knowledge, or the maximum step limit is reached.

The output of AAIE consists of multi-turn question-answer pairs that include the original conversation, the imagined conversation, and question-answer data of each evolution step. Further, some concepts were originally not in the context, but through this method, they have been added to the context. Thus it is easier for LLM model to understand a humor's rationale.

2. For the guided explorative self-improvement tuning, why do we need r+ and r-? do we use them in the fine-tuning?

The $r^+$ and $r^-$ for online DPO training are helpful to enhance the humorous reasoning path and weaken inhumorous reasoning path. It is helpful to simulate the human thinking process. Therefore, $r^+$ and $r^-$ is necessary to realize the reasoning process. We give an example of $r$ in Figure 12 in Appendix (we modify it in line 886 of revised paper)..

In addition, we do not use it in the fist stage i.e. the fine-tuning. Because it is more efficient to train model's preference on DPO training stage.

3. Failure showcase and analysis.

There are some comparative data. Column 2 is the response from CLoST and Column 3 is a better answer by human rating (we modify it in line 1016 of revised paper).

The creativity is uneven, and the creativity shown in the samples of training dataset varies widely. For example, in case 1, the response from CLoST is an internet meme and the response from better one using a Chinese proverb to compare. In terms of creativity, the latter have broader hands. In case 2, a shorter and antithetical answer is more witty. There are no failure examples, only in some people's preferences, it is not humorous.

4.Human evaluation for generated humor quality.

Thanks a lot for your suggestion. We choose the first 200 samples in validate split of Ruozhiba dataset\footnote{https://github.com/Leymore/ruozhiba/tree/main?tab=readme-ov-file}. And we call GPT-4o to rewrite the Ruozhiba query into a question and answer pair, placing the punchline in the answer section. Then four LLMs generate responses to each question as four options. And Then we conduct a user preference study to directly verify the creativity of LLMs. We present question and several corresponding replies, and ask users to choose the most creative and humorous responses as shown in . Here we select four advanced LLMs to generate responses for a total of 200 questions, and the four responses from four distinct LLMs are randomly permuted in options. We conduct an extensive survey through the online survey platform\footnote{https://www.wjx.cn/}, ultimately collecting 15 valid questionnaires with 3000 votes. Within these collected questionnaires, we can calculate the proportion of times each LLM is selected for each question. Finally, we aggregate the total number of times each LLM is chosen across all validate samples as shown in Figure 6(c). The ratio of this sum to the overall number of selections among all LLMs signifies the user preference for each LLM. Also we calculate the win rate based on the dimension of the problem as shown in Figure 6(b). (we modify it in line 430 of revised paper).

Dear Reviewer JsCo,

I hope this message find you well. We have carefully considered your feedback and have made significant improvements to the manuscript. We truly value your insights, and your expertise has greatly contributed to enhancing the quality of our work. Could you please let us know if the revisions meet your expectations?  As the deadline for discussion nears, we kindly ask if you could review our updated paper. We are eager to address any additional questions that may arise. Thank you for your invaluable support and consideration.

Sincerely,

Authors

Because the field of jokes is quite new, we appreciate your feedback so that we can better express our innovation.

1. Question about not breaking the shackles of pattern recognition.

And it is true that pattern recognition is an unavoidable for most work related to deep neural networks. As for the process of humor generation, human preference is an unavoidable problem, but beyond human preference, humor generation has its reasoning logic. The highlight of our method is finding a way to accomplish such a special reasoning process.

Humor reasoning is a multi-hop process, and each hop is based on external knowledge injection and proper rationales. Without them, it is difficult for the model to understand the internal humorous logic, making it prone to pattern recognition. Therefore, AAIE is proposed to inject and augment knowledge into the original training data. It will be facilitative for LLMs to understand the underlying logic and rationale. And then in the stage of GESIT, the reasoning logic for each online generated response is extracted using GPT-4o. In this process, external knowledge is introduced again to assist the model in logic reasoning and human preferences learning. Experimental results demonstrate that the combination of these two methods can both enhance the model's judgment ability and improve its generative capability.

For example, there is a conversation, “What do you think of Aeroflot?” “I was on this airline once, and arrived an hour early, and finally I have been divorced three years.”. Aeroflot has nothing to do with divorce seemingly. But they could be related by this way, "Aeroflot $\rightarrow$ Well-known feature is fast $\rightarrow$ It could arrive an hour early $\rightarrow$ It leads to trip disrupted $\rightarrow$ So something to ruin the marriage was found $\rightarrow$ Finally the responder is divorced." From the example, we can conclude that there is a sparse knowledge graph between question and answer. The key of making a creative leap is to mitigate the information insufficiency issue. To realize it, the injection of knowledge beyond the literal is necessary to reasoning, which is the goal of CLoST.

2. AAIE's necessity, and ablation experiments on AAIE and analysis.

We supplement ablation study in Table 3 and 4 (we modify it in line 379, 447 of revised paper).. Row 5-6 shows that model trained with AAIE realizes a noticeable improvement. Row 1 in Table 3 is the performance of QWEN-1.5-32B. Row 2-4 show the performance of gradually adding method on Oogiri-GO-en dataset. The results show that with the increase of tasks, especially in the teacher-student system, the performance of judgment is improved. AAIE with Oogiri-GO-en only makes the performance decrease little. It may be cause by overfitting to the divergence of thought on Oogiri-GO-en. In addition, the DAT test is conducted on ablation study in Figure 5(b), which shows that AAIE enhance the divergent associate thinking ability.

3. Experiments set up on selection and parameters set up.

Thanks for your suggestion. With respect to the multiple-choices questions, there are varying difficulty levels in the dataset, and its variety is primarily reflected in the choice options. We collect data from varied humor generation games such as Origi-GO. In these games, a question gains many responses from human and these responses are ranking by human voting. We select response with different votes to construct choices. A simple case consists of responses with voting numbers that differ significantly, while a hard case consists of responses with voting numbers that differ marginally. And with increase of options' number, the votes number difference between options will shrink. Finally, we clarify that we randomly shuffle the options in training and validate sets.

Training pipeline details are supplemented in Appendix in (we modify it in line 783 of revised paper). and Experiments details used in generation are list in Table 9 (we modify it in line 791 of revised paper). More parameters used in generation are list in Table 9.

4. Quotation error.

I am so sorry for the mistakes and I modify them in the revised version.

Dear Area Chair gsN1,

To facilitate the reviewers' experience and verification of the creativity of the model, our weights and inference code are currently ready and prepared to be made public. However, due to the double-blind reason, we prepared an anonymous GitHub repository but encountered file size limit when committing model weights to GitHub. Since there is no anonymous mode on huggingface, which way would you recommend us sharing the materials while following the double blind policy?

Sincerely,

Authors

1. Comparison on Chinese benchmarks.

In the benchmark, a question gains many responses from human and these responses are ranking by human voting. We select response with different votes to construct choices. A simple case consists of responses with voting numbers that differ significantly, while a hard case consists of responses with voting numbers that differ marginally. The results show that CLoST still gain the state-of-the-art performance.

2. Comparison on Ruozhiba dataset.

we supplemented an experiment on the Ruozhiba dataset, which most well-known LLMs have been trained on. We asked GPT-4o to rewrite the Ruozhiba query into a question and answer pair, placing the punchline in the answer section. Then, we asked GPT-4o again to rewrite the ground-truth answer into a non-humorous version. Based on the positive-negative pair data, LLMs were tested, and the results are shown in Table 9 (we modify it in line 1034 of revised paper). The results show that CLoST also realizes state-of-the-art performance on Ruozhiba dataset.

3. Comparison on English Benchmarks.

We collect data from varied humor generation games such as Origi-GO. In these games, a question gains many responses from human and these responses are ranking by human voting. We select response with different votes to construct choices. A simple case consists of responses with voting numbers that differ significantly, while a hard case consists of responses with voting numbers that differ marginally. And with increase of options' number, the votes number difference between options will shrink. Finally, we clarify that we randomly shuffle the options in training and validate sets. CLoST also is the SOTA method on these English benchmarks.