A Large-scale Dataset with Behavior, Attributes, and Content of Mobile Short-video Platform

Q1: The paper does not compare model performance trained on this new dataset with previous benchmarks in the literature, making it challenging to assess the dataset’s practical utility. This absence of baseline comparisons limits our ability to measure performance gaps and evaluate how well (or poorly) this dataset performs relative to established standards. For me, this is the most critical aspect, as such a comparison is essential to determine the dataset's true impact and value in advancing the field.

Response: We have supplemented experimental results of 8 recommendation algorithms (including both general and multimodal recommendation methods) to validate the practical usage of our dataset on user modeling. The results are as follows:

Q2: The selection criteria for users and videos are not explicitly detailed, raising concerns about the dataset's representativeness and potential sampling biases. The paper does not also sufficiently analyze or acknowledge potential biases, such as volunteer self-selection bias, geographic concentration, or device type distribution, which could affect the dataset's generalizability.

Response:

We have conducted the validation of the bias issue on two aspects.

• Validation of the macroscopic data distribution. We have checked the consistency between the user-side data of our dataset and the public information released by the short-video platform. According to the official report, there were users with 44% female and 56% male. In our dataset, there are 43% female and 57% male, which is consistent with the real user distribution on the platform without obvious bias. This provides evidence for the reliability of the overall collected data, and thus other user-side attributes can also be ensured unbiased.
• Validation of the microscopic specific data. We have addressed the bias issue of specific user-side data on two aspects. On the one hand, we have introduced internal consistency checking in the survey to estimate how consistent the results are for different questions about the same fact. If volunteers were to randomly report their information, the probability of passing the internal consistency check is extremely low. While actually we find no obvious failure of the internal consistency checking during the data collection, meaning that the self-reporting data is actually accurate. On the other hand, we have compared the self-reporting data with the public user profile on the platform uploaded by themselves (e.g., gender, age) and information recorded by the proxy agent we use (e.g., device). We found that there's nearly no conflicting information, meaning that these data is originally unbiased. These treatments have already shown the overall reliability of users' self-reporting data. Besides, we have further filtered the very few unreliable data to keep the dataset unbiased.

Moreover, considering that the volunteer group is large enough and covers a wide range of user attributes, the experimental group can be already representative. Therefore, the representativeness of the dataset is not a big issue.

Q5: Could you provide baseline experiments demonstrating the dataset's applicability and effectiveness for common tasks such as recommendation, user modeling, or video classification?

Response: We have supplemented experimental results of 8 recommendation algorithms, please see the response to Q1.

Q6: Expand the literature review to encompass the latest publicly available short-video datasets. Provide a comparative table highlighting key features, such as the number of users, videos, interaction types, and content richness, to clearly demonstrate how your dataset stands out.

Response: Please see the response to Q4.

Q7: Please adjust Figure 4, as the text appears too close together. Changing the format would improve its readability.

Response: Thanks for the suggestion and we have corrected that in the revised paper.

Q8: It would be helpful to enlarge the legends in Figures 3, 4, and 5, as they are currently quite small. Also, please consider using a color palette that is accessible to colorblind viewers, including in Figure 6.

Response: Thanks for the suggestion and we have corrected that in the revised paper.

Q9: Include case studies or illustrative experiments that showcase the dataset's application in areas such as user modeling, fairness analysis, or studying filter bubbles. Presenting preliminary results or hypothetical scenarios can help researchers envision practical uses and inspire innovative applications.

Response: Thanks for the suggestion. We have supplemented experimental results of 8 recommendation algorithms to validate the practical usage of our dataset on user modeling. The results are shown in the response to Q1.

As for social science and human behavior understanding research, we provide some preliminary ideas for studying filter bubble phenomenon with our dataset as an example. The filter bubble refers to the situation where recommendation algorithms overly tailor content to a user's preference, resulting in limited content exposure for the user on the platform. To quantify content coverage, our dataset includes a hierarchical content category system. Coverage at a category level c is defined as

$$\frac{N_{seen}(u,c)}{N_{all}(c)}$$

where N_seen represents the number of level-c categories a user has interacted with over a period, and N_all is the total number of level-c categories available. A user's filter bubble extent can then be determined by comparing this metric against a threshold, such as the average across all users. Subsequent analyses could investigate how user behaviors, such as viewing time and content preferences, influence the severity of filter bubbles across different category levels. Additionally, tracking the temporal evolution of filter bubbles at each level could provide insights into their dynamics and persistence over time.

Q10: Describe the handling of multilingual content within the dataset. Specify the languages supported by the ASR system and outline any post-processing steps taken to enhance transcription accuracy. Provide statistics on the accuracy rates or error rates of the ASR results, if available.

Response: We utilized an advanced model SenceVoice-Small for speech recognition and LLaMA-3-8B for translation, obtaining bilingual (Chinese and English) ASR texts. We further conduct manual examination and refinement of the obtained text to ensure the data quality.

Q11: How do the potential biases affect the dataset's generalizability, and what steps have been taken to mitigate them?

Response: Please refer to the response to Q2.

Q12: Add a dedicated section to discuss the dataset's limitations, covering potential biases, data sparsity in specific interaction types, and gaps in demographic coverage. Transparently addressing these issues helps researchers to consider them in their analyses and supports trust in the dataset's integrity.

Response: We discuss some limitations of the dataset as follows, which is also supplemented in Section A.6 of the Appendix. First, the dataset's current scale is smaller compared to some existing recommendation datasets that lack comprehensive attribute and content data. This limitation arises due to the challenges of collecting data from volunteers. Second, the dataset primarily focuses on users from a single country. Expanding its coverage to include users from multiple countries would further enhance its generalizability. Third, the amount of explicit user feedback is limited, as the data was collected over a relatively short period. In the future, we will extend the scale and user coverage of our dataset and update it timely.

Q3: Apart from qualitative t-SNE visualizations, the study lacks a quantitative evaluation, such as clustering metrics or downstream task performance, to effectively validate the quality of the extracted video features.

Response: We have tested 4 typical multimodal recommendation methods with and without video features to validate the practical usage of these features. From the results, it can be seen that there's common performance drop without video features of all methods, validating the quality of extracted video features.

Q4: The review of existing datasets and related work is not exhaustive, potentially overlooking relevant studies and resources that could contextualize the dataset's contributions.

Response:

We have provided a comparison table about relevant short-video datastes as follows. In summary, a core limitation of existing datasets is the limited content richness. Microvideo-1.7M [1] provides only video category information, while datasets like KuaiRec [2], REASONER [3] and Tenrec [4] include only video text data or pre-extracted visual features from video thumbnails. Although the more recent Microlens [5] dataset includes raw videos, it is constrained by a limited number of videos. More critically, it lacks rich user-side information, such as demographic and geographic attributes, offering only user ID data. By comparison, our dataset is more comprehensive and balanced on content richness and data scale.

[1] Chen, Xusong, et al. "Temporal hierarchical attention at category-and item-level for micro-video click-through prediction." Proceedings of the 26th ACM international conference on Multimedia. 2018.

[2] Gao, Chongming, et al. "KuaiRec: A fully-observed dataset and insights for evaluating recommender systems." Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022.

[3] Chen, Xu, et al. "REASONER: an explainable recommendation dataset with comprehensive labeling ground truths." NIPS 2024.

[4] Yuan, Guanghu, et al. "Tenrec: A large-scale multipurpose benchmark dataset for recommender systems." NIPS 2022.

[5] Ni, Yongxin, et al. "A content-driven micro-video recommendation dataset at scale." arXiv preprint arXiv:2309.15379 (2023).

Q1: The link to the dataset seems to be broken, possibly due to a network issue on my end. Could you provide guidance on accessing it? I may consider raising the score if access is successful.

Response: Sorry for the problem. We now provide another link for your review: https://www.dropbox.com/scl/fo/5z7pwp6xjkrr1926vreu6/AFYter5C6BDTOCpxkxF0k9Y?rlkey=p28j6u1fl1ubb7bufiq16onbl&st=7aktff85&dl=0, in which we sample 10 video files with corresponding title text, ASR text, visual feature data and the full interaction data. Details about the shown data are as follows:

• raw_file: the folder containing original video files with the format "video_id.mp4".

• video_feature_total: the folder containing all extracted visual features with the format "video_id.npy".

• interaciton.csv: the file containing all behavior data and attribute data.

• asr_zn: ASR text in Chinese of each video with the format "video_id.txt".

• asr_en: ASR text in English of each video with the format "video_id.txt".

• title_en: video title text in English with the format "video_id.txt".

• categories_cn_en.csv: the name of categories in English and Chinese.

Q2: From a data perspective: Since the data originates from a single social media platform, user behavior is influenced by the platform’s

• recommendation algorithm. This introduces significant bias due to these recommendation effects.
• The visual features were extracted using a 2016 image model, while more advanced models are now available that can extract features from video encodings. Thus, the extracted visual features may be insufficient.

Response:

About the bias: We agree that exposure bias is an inherent challenge in most offline recommendation system datasets, including our dataset. This bias arises naturally from real-world application scenarios where users are exposed to limited content. As such, all datasets with actual user interactions are likely to exhibit some degree of exposure bias, it mirrors the conditions under which recommendation systems operate in practice. Therefore, addressing exposure bias is not merely a matter of dataset but requires more effort from advanced recommendation algorithms (e.g., some causal inference methods).

About the visual features: We have used a more advanced model ViT-base for visual feature extraction, and the files are updated.

Q3: From a validation perspective:

• The conclusions presented in Section 3.1.2 lack depth and seem self-evident.
• The paper lacks proof-of-concept experiments to demonstrate the dataset’s value in the proposed research areas, which somewhat diminishes the paper’s impact.
• Most validations rely on statistical analyses, rather than truly testing the representational ability of the video data. Statistical data alone is insufficient to prove the dataset’s research value.

Response:

About analysis in Section 3.1.2: We have added more detailed analysis about relations between different user behaviors, please refer to Section 3.1.2 in the updated paper.

About proof-of-concept experiments: We have added experiments of recommendation algorithms and preliminary ideas for studying filter bubble with our dataset, please refer to the response to Q4 for more details.

About validation of the representation ability of the video data: We have tested 4 typical multimodal recommendation methods with and without video features to validate the practical usage of these features. From the results, it can be seen that there's common performance drop without video features of all methods, demonstrating that the value of extracted video features.

Q4: Overall, while the dataset provides fresh data and unique insights into user actions, the paper lacks experimental results demonstrating the dataset’s applicability across the various domains it proposes.

Response: We have supplemented experimental results of 8 recommendation algorithms (including both general and multimodal recommendation methods) to validate the practical usage of our dataset on user modeling. The results are as follows:

As for social science and human behavior understanding research, we provide some preliminary ideas for studying filter bubble phenomenon with our dataset as an example. The filter bubble refers to the situation where recommendation algorithms overly tailor content to a user's preference, resulting in limited content exposure for the user on the platform. To quantify content coverage, our dataset includes a hierarchical content category system. Coverage at a category level c is defined as

$$\frac{N_{seen}(u,c)}{N_{all}(c)}$$

where N_seen represents the number of level-c categories a user has interacted with over a period, and N_all is the total number of level-c categories available. A user's filter bubble extent can then be determined by comparing this metric against a threshold, such as the average across all users. Subsequent analyses could investigate how user behaviors, such as viewing time and content preferences, influence the severity of filter bubbles across different category levels. Additionally, tracking the temporal evolution of filter bubbles at each level could provide insights into their dynamics and persistence over time.

Thank you for your effort and reply!

---

For Q1: The update of the dataset link has resolved my concern.

---

For Q2: My concern has not been addressed. Since the goal is to study user behavior while watching videos on social media, it is crucial to consider that most users typically use multiple platforms. Sampling data from multiple platforms would effectively mitigate such biases. I believe this type of bias is critical when constructing a user behavior dataset. Research based on biased data can lead to conclusions that are limited by the constraints of a single platform.

---

For Q3:

1. In the updated version of the paper, I did not find Section 3.1.2. Additionally, if modifications were made, it would be better to highlight them in a different color for clarity.
2. Response in Q4 rebuttal.
3. I find the comparison insufficient. I believe the dataset’s proposed video representations should be compared with other video representations to demonstrate their role and distinctions better.

---

For Q4:

1. The result of the recommendation system is a good example to help me understand the value of your data.
2. Since the formula you proposed makes sense, why not conduct some experiments based on your datasets and analysis instead of merely presenting assumptions?

---

Although additional experimental results have been included, the relevance of these results to the dataset and the validation of its value remains very limited.

Q6: Figure can be moves so they are closer to the place they are mentioned. Right now Figures 3 and 4, are like 2 to 3 pages away from the paragraph they are mentioned in. It would be better if authors manage to arrange them in a way that they are closer to the sections they belong to.

Response: Thanks for the suggestion and we have corrected that in the revised paper.

Q7: What are the implications on the fact that most of the users from the dataset come from a single country? What are the benefits from it? What are the downsides of having so little diversity in terms of country of origin? Could the conclusions made with this data be extrapolated to other places? If not, what could be the solution for this?

Response: Please see the response to Q1.

Q8: What was the criteria to pick the classes for Category I?

Response: Videos were first clustered based on similarities in their title and tag texts. Each cluster was manually reviewed to define the overarching theme it represented, leveraging human expertise to finalize the content of Category I. To ensure accuracy, every video was manually checked to confirm that its assigned category matched its actual content, with misclassified videos corrected.

Q9: What was then the Criteria to make the hierarchies from I to II and III?

Response: It's also based on video clustering and human experience to form these categories, see the response to Q1 for more detail.

Q10: Although the authors have been very transparent about their data collection and statistics. I still think that a closer look to the data release and the biases on the dataset should be checked. I am no expert in these topics and from my perspective looks like the authors have been transparent and have everything in order for the release of the data. However, I still think that a closer look into it would be beneficial given that the dataset might contain sensitive information about the users involved and their preferences and interactions with the platform.

Response: We would like to further clarify the user information in our dataset. All the participants have signed the informed consent form and agreed their anonymized information to be used for research purposes. For each participant, we collected seven pieces of information: user ID, gender, age, device price, name of the residential city, city's level, and community type. We provide a specific example in the following table. The user ID has been hashed, which was mapped to a number between 1 and 10,000 to prevent the identification of users. For other demographic and geographic attributes, we only collected coarse-grained data, for example, the geographic granularity is limited to the city level. Such information is not sufficient to be utilized for the identification of users.

Q1: Every dataset has biases. However, this one in particular has a very critical one which is the population that it focused on. In Figure 5d, it's clear that the users that appear in the dataset are mostly from China, which makes the dataset tailored to a single demographic population. Although, the data is still valuable, conclusions that will be made in the future using this data can be only applied to the Chinese population. I would like to know what to the authors and other reviewers think about this critical point.

Response: Collecting a comprehensive dataset of short-video platforms with behavior, attributes and content data from volunteers is actually very challenging, even within a single country. Although the population are most Chinese, our dataset has covered diverse user groups in terms of demographic and geographic attributes, as detailed in Section 3.2 and Figure 5 in the main text. Therefore, the data is already rich and diverse, and the generalization ability can be ensured to some extent. Moreover, it is important to note that many existing datasets, such as Kuairec [1], REASONER [2], Tenrec [3] and Microlens [4], similarly focus on a single platform and primarily cover users from one country. This is often due to the practical challenges of cross-regional data collection. We are actively working to expand our data collection efforts to include users from other countries to further enhance the population coverage.

[1] Gao, et al. "KuaiRec: A fully-observed dataset and insights for evaluating recommender systems", CIKM 2022.

[2] Xu, et al. "REASONER: An Explainable Recommendation Dataset with Multi-aspect Real User Labeled Ground Truths Towards more Measurable Explainable Recommendation", arXiv 2023.

[3] Yuan, et al. "Tenrec: A large-scale multipurpose benchmark dataset for recommender systems." NeurIPS 2022.

[4] Ni, Yongxin, et al. "A content-driven micro-video recommendation dataset at scale." arXiv 2023.

Q2: Most of the data provided by the data is simply collected by how the users interact with the platform. However, there are a couple of design choices that were unclear to me:

• There is no explanation on how the video categories I,II, and III were chosen. The authors mention that they are hierarchical, which sounds like a good idea. However, the is no explanation on how the hierarchies were chosen and how were the classes picked from the tags, titles, and content of the videos. How was this discretization of classes done for I? How was then the hierarchical structure form to have II and III?
• It was not clear to me what is the "effective view" label, what does it try to convey and why is the threshold of 3 seconds picked?

Response:

The process for defining the video categories (I, II, and III) was carried out iteratively as follows:

Category I Formation: Videos were first clustered based on similarities in their title and tag texts. Each cluster was manually reviewed to define the overarching theme it represented, leveraging human expertise to finalize the primary categories. To ensure accuracy, every video was manually checked to confirm that its assigned category matched its actual content, with misclassified videos corrected.

Category II Formation: For each cluster within Category I, the process above was repeated. Videos in each primary category were further clustered, manually reviewed, and refined to define the secondary subcategories.

Category III Formation: The same procedure was applied to each secondary category, further refining the classification to form tertiary categories.

About "effective view": Users passively engage with recommended videos on the platform, and they may quickly skip over certain videos, especially those they find uninteresting. This label provides an implicit metric to measure users' genuine attitudes toward the content they watch. As for the 3 seconds threshold, it's based on our observation and experience. Considering that the dataset includes original watching time, this threshold can be adjusted according to the real application scenario.

Q3: Whenever not sure about the gender of the referring person, it's better to use THEY as pronoun instead of saying he/she all the time. It also easier to read this way.

Response: Thanks for the suggestion and we have corrected that in the revised paper.

Q4: Figure 4 x axis should number of likes in plural.

Response: Thanks for the suggestion and we have corrected that in the revised paper.

Q5: Figure 4 caption can be further improved by stating that the data presented is per user not per video, it can be confusing if one reads the figure before reaching the paragraph in which it is mentioned.

Response: Thanks for the suggestion and we have corrected that in the revised paper.

Q1: The dataset proposed in the paper lacks testing in tasks of different research areas (such as user modeling, social sciences, and understanding human behavior) mentioned in the paper, which results in insufficient clarity regarding its practicality and applicability.

Response:

Thanks for the suggestion. We have supplemented experimental results of 8 recommendation algorithms (including both general and multimodal recommendation methods) to validate the practical usage of our dataset on user modeling. The results are as follows:

As for social science and human behavior understanding research, we provide some preliminary ideas for studying filter bubble phenomenon with our dataset as an example. The filter bubble refers to the situation where recommendation algorithms overly tailor content to a user's preference, resulting in limited content exposure for the user on the platform. To quantify content coverage, our dataset includes a hierarchical content category system. Coverage at a category level c is defined as

$$\frac{N_{seen}(u,c)}{N_{all}(c)}$$

where N_seen represents the number of level-c categories a user has interacted with over a period, and N_all is the total number of level-c categories available. A user's filter bubble extent can then be determined by comparing this metric against a threshold, such as the average across all users. Subsequent analyses could investigate how user behaviors, such as viewing time and content preferences, influence the severity of filter bubbles across different category levels. Additionally, tracking the temporal evolution of filter bubbles at each level could provide insights into their dynamics and persistence over time.

Q2: Different tasks and methods have not been tested on this dataset, resulting in an incomplete benchmark. This may lead to confusion regarding the use of the dataset due to the absence of reference benchmark results.

Response: We have supplemented experimental results from nine recommendation algorithms to validate the utility of our dataset for user modeling. Additionally, we provided hypothetical use cases and research ideas on studying the filter bubble phenomenon to further demonstrate its practical applicability. For detailed explanations, please refer to the response to Q1.

Q3: Visual examples of different data such as video, images, text, and audio of behavior, attribute, and content data should be given. Both the paper and the supplementary material do not provide sufficient visual instances. It is not good for readers.

Response: We have supplemented some visual examples of video content in out dataset, including images, titles, tags, ASR texts, categories and duration in Figure 11 of the Appendix. The attribute data for both users and videos has been presented in Table 3 of the Appendix. The raw videos are listed on the website.

Q4: There are some minor grammar errors and typos

Response: Thanks for pointing out, we have further checked the grammar and typo issue, the paper has been updated.