Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior

Thank you for reviewing our paper and for your insightful questions 🙂.

Regarding the figure and table font sizes, we apologize for the inconsistency. We plan to address this by relocating some sections to the appendix and ensuring uniform font sizes throughout.

Regarding your point (1) on Behavior Finetuning:

We utilize the train subset of the CBC data for Behavior Finetuning. Here is the breakdown for different tasks using various subsets of the CBC data:

1. Behavior Simulation:

   a. YouTube Videos: 2027 videos were selected without temporal overlap with the train set, encompassing diverse brands, industries, dates, and receiver behavior.

   b. Twitter Tweets (new dataset):

   • Brand Separated: 1557 tweets from diverse brands, industries, dates, and receiver behavior.

   • Time Separated: 782 tweets without temporal overlap with the train set from diverse brands, industries, dates, and receiver behavior.

2. Content Simulation:

   a. YouTube Videos: 2027 videos chosen without temporal overlap with the train set.

   b. Twitter Tweets (new dataset):

   • Brand Separated: 1557 tweets sampled from diverse brands, industries, dates, and receiver behavior.

   • Time Separated: 782 tweets without temporal overlap with the train set from diverse brands, industries, dates, and receiver behavior.

3. Behavior Domain Adaptation Tasks:

   a. Emails Corpus: 3000 email-segment pairs without temporal overlap between train and test sets, using emails until July 2022 for training and testing on emails from August 2022 onwards.

   b. LVU Benchmark [1]: Zero-shot testing on the entire dataset without prior training.

4. Content Understanding Tasks:

   a. Advertisement Understanding Corpus (Hussain et al, 2017): Zero-shot testing on the entire dataset without training.

   b. Persuasion Strategy Corpus (Bhattacharyya et al, 2023): Zero-shot testing on the entire dataset without training.

5. Behavior Understanding Tasks:

   a. Replay Behavior Understanding: No training, 200 videos randomly sampled from the behavior simulation test set.

   b. Comment Sentiment Understanding: No training, 200 videos randomly sampled from the behavior simulation test set.

Additionally, subsequent to paper submission, we've compiled a large-scale content behavior dataset comprising images and corresponding human behavior extracted from 168 million tweets across 10135 enterprise Twitter accounts from 2007 to 2023. This dataset includes account names, tweet text, associated media verbalizations (including image captions, keywords, colors, and tones), tweet timestamps, and like counts. We plan to release both the YouTube and Twitter corpora. Our experiments with behavior simulation, domain adaptation, and content simulation on this corpus yield results consistent with those reported in the paper:

Behavior Simulation. Give content, channel, time, predict behavior (High, Low). Reported values are 2-way classification accuracies

(Brand Separated means that the train and test set don't have any overlap in terms of brands, Time Separated means that the test set starts after the last tweet in the train set)

Content Simulation Experiment. Given behavior, channel, time, tweet media caption as prompt, predict content (tweet text)

(continued in the next comment)

About inputting the visual tokens to GPT-3.5 and 4: we verbalize videos and images and feed them to GPT-3.5 and 4. We test LCBM in the same conditions and find that there is little difference in performance if visual tokens are not given.

For instance, we show results on the Replay Value Prediction task of Behavior Simulation. Similar results are observed on other tasks. Thanks for pointing this out; we will add these results to the appendix in the next version.

Additionally, we show results on the Likes-to-Views ratio task of Behavior Simulation.

Verbalization pattern of videos for the behavior understanding task, without using video tokens:

The video has the following scenes:
Scene 1: {ASR: Welcome to a quick tutorial , OCR: <product>, Captions: A desktop interface , Replays: 60},
Scene 2: {ASR: on using <product> to edit , Captions: A computer interface , with an image of a white horse. Objects − Horse, Grass,
Fence., Replays: 53}, ...
It was posted by <channel> with the title ' Using <product> like a Pro ' on Aug 15 2022. <channel> has 100k subscribers . This video was viewed by 346 thousand people and liked (as a percentage of likes /views) by 2.3% people.

[1] Chao-Yuan Wu, Philipp Krahenb, Towards Long-Form Video Understanding, CVPR 2021, Link: https://openaccess.thecvf.com/content/CVPR2021/papers/Wu_Towards_Long-Form_Video_Understanding_CVPR_2021_paper.pdf

Regarding (2) (presentation), we regret any confusion caused. Could you kindly specify the specific figures or sections that posed challenges for clarity in the upcoming version?

Re (3), while LLMs might be slow for certain specific tasks, we believe they present a more general framework, which has empirically shown much better content understanding and generation capabilities than any other systems historically. This motivates us to use LLMs (as compared to other competitive technologies) for behavior understanding (the next step and the third level of communication).

• To enable LLMs to learn behavior and content, following the T5 methodology, we convert behavior learning task to a “text-to-text” problem. We show that a model trained on ~2 trillion tokens (Llama), can be taught behavior with a few billion additional tokens. This shows that an adequately trained LLM might be taught a behavior as a new modality relatively easily.

• A similar evidence is also provided by LLMs trying to learn the vision modality, such as Llava, VideoLlama, Minigpt-4, and others [4,5,6], where they tune an LLM for vision modality with a few billion vision tokens (similar scale as us).

[1] MMLU Benchmark: https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu

[2] Winogrande Benchmark: https://paperswithcode.com/sota/common-sense-reasoning-on-winogrande

[3] HellaSwag Benchmark: https://paperswithcode.com/sota/sentence-completion-on-hellaswag

[4] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning

[5] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model

[6] Minigpt-4: Enhancing vision-language understanding with advanced large language models.

Thank you for your review and interesting questions 🙂.

Regarding (1), although we acknowledge the potential of enhancing the general reasoning ability of Language Models (LLMs) to address the issue of communication effectiveness, our study presents compelling evidence that directly exposing LLMs to behavior tokens alongside content tokens can be a more effective method to instill behavioral understanding. Our approach and underlying rationale for fine-tuning diverge significantly from established methodologies. Specifically:

1. While GPT-3.5 and GPT-4 have demonstrated superior general comprehension across various benchmarks [1,2,3], our research illustrates that training an LLM on behavior tokens enables a 13B model to surpass much larger models—despite their stronger general understanding—across all tasks. This performance extends to both tasks specifically fine-tuned and those that were not originally targeted for fine-tuning.

2. In-context learning, although impressively successful in other domains, proves ineffective in the behavior domain. Large in-context-finetuned language models (GPT-3.5, GPT-4, etc.) exhibit performance comparable to a random baseline, indicating their lack of exposure to behavior tokens.

3. Incorporating behavior tokens in content and behavior simulations not only enhances behavior and content simulation (as anticipated) but also unexpectedly improves behavior domain-adaptation (Table-6), behavior-understanding (Table 4, Figs. 7, 8), and content understanding (Table-5). Notably, LCBM exhibits enhancements in these tasks despite not being explicitly trained for them. This aligns with how LLMs, trained for next word prediction, showcase proficiency in various tasks such as question answering, summarization, and natural language inference. We demonstrate these capabilities across three behavior domain datasets (YouTube, Emails, and LVU) and five tasks spanning two content domain datasets.

4. Presently, large language models focus on training with content tokens (text, vision, videos, etc.) while disregarding behavior tokens (Biderman et al 2023, Penedo et al 2023). Consequently, LLMs lack behavior simulation (Table 1, 2), content simulation (generation) (Table 3), and behavior understanding (Table 4) capabilities.

Further, we show our results on four datasets beyond Youtube:

1. Long Video Understanding (LVU) Benchmark
2. Emails dataset
3. Two Advertisement Understanding Datasets
4. Additionally, subsequent to paper submission, we've compiled a large-scale content behavior dataset comprising images and corresponding human behavior extracted from 168 million tweets across 10135 enterprise Twitter accounts from 2007 to 2023. This dataset includes account names, tweet text, associated media verbalizations (including image captions, keywords, colors, and tones), tweet timestamps, and like counts. We plan to release both the YouTube and Twitter corpora. Our experiments with behavior simulation, domain adaptation, and content simulation on this corpus yield results consistent with those reported in the paper:

Behavior Simulation. Give content, channel, time, predict behavior (High, Low). Reported values are 2-way classification accuracies (Brand Separated means that the train and test set don't have any overlap in terms of brands, Time Separated means that the test set starts after the last tweet in the train set)

Content Simulation Experiment. Given behavior, channel, time, tweet media caption as prompt, predict content (tweet text)

(Continued in the next comment)

Thank you for your thoughtful review and interesting insights 🙂. We agree that alignment mechanisms, such as through SFT and RLHF on question-answer pairs, is a kind of behavior / intent alignment. We will add this to the paper. It would be an interesting lens for the related work.

In the limited rebuttal time, we tried varying the amount of data on some of the results, We will show more complete set of results in the next version of the paper.For instance, we show results on the Replay Value Prediction task of Behavior Simulation.

We vary our dataset size with 5k and 10k unique videos, and compare performance on the behavior simulation task. We find that increasing dataset size helps in achieving better performance.

Additionally, we show results on the Likes-to-Views ratio prediction task of Behavior Simulation.

Thanks for pointing this out, we will add more such results to the appendix in the next version.

Regarding (2) (comparison of instruction-tuned models with non-instruction-tuned ones):

• Prior research has consistently demonstrated the efficacy of in-context learning for larger models. However, our study specifically examines GPT-3.5 and GPT-4 over behavior-related tasks, revealing an incongruity. Notably, the performance variation between 3, 5, and 10 shots is statistically insignificant, indicating a lack of exposure to behavior tokens within these models.

• Furthermore, the stark performance contrast between a 13B behavior finetuned model and significantly larger in-context-finetuned language models (such as GPT-3.5, GPT-4, etc.) is notably high, exceeding 300% (as illustrated in Table 1, 2) and akin to, or even worse than, random performance.

• Importantly, this discrepancy persists even when LCBM is tested in settings disparate from its training data, notably in behavior understanding and behavior domain-adaptation tasks. This highlights the imperative need for LLMs, trained primarily on content tokens (text, vision, videos, etc.), to integrate the knowledge of behavior tokens for comprehensive understanding.

References:

• Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023.

• Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile. arXiv preprint arXiv:2201.07311, 2022

Thank you for your insightful review! 🙂

About (1) technique novelty, we agree with your review. Other than showing how to do behavior finetuning (using content and behavior instruction tokens), we do not intend to go for any other technique novelty in this paper because we find existing techniques are good enough to train our model well. Rather, we focus on the following contributions:

1. Incorporating behavior tokens in content and behavior simulations not only enhances behavior and content simulation (as anticipated) but also unexpectedly improves behavior domain-adaptation (Table-6), behavior-understanding (Table 4, Figs. 7, 8), and content understanding (Table-5). Notably, LCBM exhibits enhancements in these tasks despite not being explicitly trained for them. This aligns with how LLMs, trained for next word prediction, showcase proficiency in various tasks such as question answering, summarization, and natural language inference. We demonstrate these capabilities across three behavior domain datasets (YouTube, Emails, and LVU) and five tasks spanning two content domain datasets.

2. Presently, large language models focus on training with content tokens (text, vision, videos, etc.) while disregarding behavior tokens (Biderman et al 2023, Penedo et al 2023). Consequently, LLMs lack behavior simulation (Table 1, 2), content simulation (generation) (Table 3), and behavior understanding (Table 4) capabilities.

3. In-context learning, although successful in other domains, proves ineffective in the behavior domain. Large in-context-finetuned language models (GPT-3.5, GPT-4, etc.) exhibit performance comparable to a random baseline, indicating their lack of exposure to behavior tokens.

4. Next, to spur research on the topic of content and behavior models, we release behavior and content tokens collected over 40,000 public domain YouTube videos containing a variety of behavior tokens.

5. Additionally, subsequent to paper submission, we've compiled a large-scale content behavior dataset comprising images and corresponding human behavior extracted from 168 million tweets across 10135 enterprise Twitter accounts from 2007 to 2023. This dataset includes account names, tweet text, associated media verbalizations (including image captions, keywords, colors, and tones), tweet timestamps, and like counts. We plan to release both the YouTube and Twitter corpora. Our experiments with behavior simulation, domain adaptation, and content simulation on this corpus yield results consistent with those reported in the paper:

Behavior Simulation. Give content, channel, time, predict behavior (High, Low). Reported values are 2-way classification accuracies

(Brand Separated means that the train and test set don't have any overlap in terms of brands, Time Separated means that the test set starts after the last tweet in the train set)

Content Simulation Experiment. Given behavior, channel, time, tweet media caption as prompt, predict content (tweet text)

The question about detailed information on how the visual encoder handles video input. We apologize for not having a clear enough coverage of this topic. We will make sure to cover this in the next version. The details are: The pipeline uses the pre-trained EVA-ViT-G with the Global Multi-Head Relation Aggregator (GMHRA) which is a temporal modeling module. For token projection, we emply a pretrained QFormer with extra linear projection to incorporate additional query tokens to include video context for training the LLM. While training, the projected video encodings are used to represent video context in the training corpus.

(continued in the next comment)