SNS-Bench: Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services

R4.Q1: Broader Social Networking Context

Thank you for the insightful observation. The construction does incorporate key social interactions:

1. Data Collection Pipeline (Section 3.2)

   • User engagement metrics reflecting community response
   • Note categories and tags chosen by creators based on audience reception
   • Comment sections capturing direct user interactions

2. Example: Note-Hashtag

   • Candidate tags include the original creator’s selections (personal preference)
   • Community-generated popular tags for similar content (crowd preference)

R4.Q2: Personalization

We give the example on personalization:

{"content":"Note Title: 108 Exercise + Reading Classics\nNote Content: Since discovering the Seed Law in October 2023, I’ve practiced 108 exercises and recited classics for over 7 months. My health has improved—better complexion, fewer wrinkles, and improved sleep. I've also experienced increased energy and financial gains.\nAs Seed Dad says, only action brings results.\nI hope everyone persists—each breakthrough leads to a better self!\nEvery drop of sweat is proof of our transformation!","candidates":"Enhance Energy,Grapefruit Jasmine,Must Visit Xinjiang,Cat Internal Deworming,Physics Tutoring for International Students,Stele Forest Museum,Sydney Spa,Buccellati Gardenia,108 Exercise,Tongue Piercing,Seed Law,Sharing What I Find Interesting,Women's Growth,Guojin Building","answer":"Seed Law,108 Exercise,Enhance Energy,Women's Growth"}

1. The tags ("Seed Law", "108 Exercise") are derived from:
   • The author's consistent personal practice (7 months)
   • Tangible health benefits (specific improvements listed)
   • Community validation (implied by their selection)
2. The selected tags represent:
   • A personal transformation journey ("Women's Growth")
   • Proven techniques with real impact ("108 Exercise")
   • A philosophical alignment ("Seed Law")
3. This demonstrates how we capture personalization:
   • Creator-defined preferences (original tags)
   • Community-endorsed interests (popular tags)

R4.Q3: Platform limitations We appreciate the concern about REDnote's note format. In our ongoing work (SNS-Bench-V2), we are addressing this through:

1. Multi-platform expansion:

• Collecting data from Twitter/X and Instagram.
• Designing unified task formulations that work across platforms.

2. Beyond note content:

• Incorporating threaded conversations.
• User history.
• Cross-post interactions.

R4.Q4: Visual loss in OCR We take the visual element concern seriously and have implemented:

1. Strict quality control:

• Human verification for all OCR conversions (Appendix B)
• Automatic filtering of low-confidence OCR results

R4.Q5: Typo and Tasks We will fix the typos and rewrite the detailed definition of tasks.

R4.Q6: Results on Chinese Version Most cases are English, with Chinese translated into English (GLM4 + manual review). To provide more results, we have translated all English cases into Chinese (GLM4 + manual review). We will release the Chinese data. The average Chinese results:

(PS. Llama-3.2-3B-Instruct and Phi-4-14B with a performance drop.)

R4.Q7: Social Networking & Personalization SNSbench captures social networking dynamics through:

1. Interaction Signals: Tasks use data with implicit social context—comments, user-generated tags, and replies. For example, Note-CHLW identifies highlight words from actual discussions, mirroring how platforms detect trending topics.
2. Personalization: Most tasks reflect personalized SNS behaviors, for example:

• Note-QueryGen: Generated queries mimic how users personalize searches based on interests (e.g., converting a skincare note into "best vitamin C serums for sensitive skin").

We kindly invite you to review our responses and reconsider your assessments. Thank you for your time and consideration!

R3.Q1: Multimodal limitation. We sincerely appreciate the reviewer’s constructive feedback. The reviewer is absolutely right to highlight the importance of multimodal interactions in modern SNS platforms. In fact, we are already working on SNS-Bench-V2, which will incorporate image-text pairs (e.g., OCR-extracted text from note images, user-generated captions) and multimodal tasks (e.g., visual hashtag recommendation, sentiment analysis). This extension will align with real-world SNS content while maintaining our focus on nuanced social understanding. We will explicitly discuss this direction in the revised manuscript’s Broader Impact section.

R3.Q2: Deeper analysis and insights. Thank you for prompting us to clarify our findings. Beyond aggregated results (Table 2), our analysis reveals critical task-specific and generalizable insights about LLMs’ social capabilities:

1. General Insights:

Even state-of-the-art models struggle with SNS-unique demands:

• Creativity: Underperform in Note-QueryGen (lexical diversity) and Note-CHLW (novelty detection) due to rigid training objectives.
• Depth: Struggle with Note-MRC (Complex) (evidence extraction) and Note-QueryCorr (Topic) (intent granularity), revealing gaps in social context modeling (Section 5.1).

2. More Task-Specific Insights:

• Note-Taxonomy: Small models (e.g., Qwen-1.5B) favor single-choice tasks (accuracy: 27.5%), while larger models (e.g., Llama3-70B) excel in multi-hop reasoning (65.12%), proving scale aids hierarchical reasoning (Table 2).
• Note-Hashtag: Most models (32B+) perform better on single-choice (Qwen-72B: 86.25%) than multi-choice (84.60%), except Claude-3.5, which leads in multi-choice (88.58%), suggesting superior multi-label adaptability (Figure 9).
• Note-MRC (Simple): Gemini/Qwen excel at binary relevance judgment (F1: ~90%) but fail to precisely extract answers (BLEU: ~55%), whereas DeepSeek-V3 achieves the best content extraction (ROUGE-L: 74.81%), highlighting a trade-off between relevance judgment and granular retrieval (Table 5).

These observations underscore that SNS challenges require both social and technical innovation—a theme we will emphasize in Section 6. We will also add a new subsection (5.3 Model Behavior Analysis) to consolidate these insights with supporting visualizations (e.g., confusion matrices for Hashtag tasks).

Response to Review 2: Thank you for your insightful feedback.

R2.Q1: Isolated evaluation of SNS tasks. We acknowledge the concern regarding task isolation in SNS-Bench and appreciate the opportunity to clarify our design rationale.

1. Task-Specific Evaluation Necessity: Social networking services (SNS) involve diverse interactions (e.g., content comprehension, sentiment analysis, recommendation), each requiring distinct capabilities. Isolated evaluation allows us to:

• Pinpoint strengths/weaknesses: For instance, models may excel at hashtag selection (structured tasks) but struggle with complex reasoning in Note-MRC (Figure 7).
• Guide targeted improvements: Task-specific metrics (e.g., F1 for Note-NER, ANLS for Note-QueryGen) reveal granular performance gaps (Table 5).

2. Holistic Insights via Aggregation: While tasks are evaluated independently, our analysis synthesizes cross-task patterns (Section 4.3). For example:

• Closed-source models consistently outperform open-source ones (Table 2), suggesting general superiority in SNS contexts.
• Tasks requiring emotional/cultural understanding (Note-Gender) exhibit higher variance, highlighting challenges in social nuance (Section 5.1).

3. Real-World Generalization: The benchmark’s diversity—8 tasks spanning 6,658 questions with varied formats (Table 1)—ensures broad coverage of SNS scenarios. By testing isolated but representative tasks, we simulate the multifaceted demands of real-world platforms (e.g., handling both classification and generation). We agree that future work could explore interdependencies between tasks (e.g., joint training). However, our current design prioritizes diagnostic clarity, enabling actionable insights for model development. Thank you again for your valuable advice. We will emphasize this rationale in the revised manuscript.

R2.Q2: Lack of creativity in evaluation framework.

We appreciate the insightful feedback. While our benchmark adopts standard evaluation metrics for individual tasks, its core value lies in the authenticity and diversity of real-world SNS scenarios. Unlike synthetic or simplified datasets, SNS-Bench captures nuanced user behaviors (e.g., informal language, cultural references) and task interdependencies (e.g., sentiment influencing recommendations) from actual SNS platforms. This fidelity enables a more grounded assessment of LLMs’ practical utility in social contexts. Future work will build on this foundation to incorporate interactive and adversarial dynamics.

R2.Q3: Citation and figure readability issues.

Thank you for your careful review. We will address both points in the revised manuscript:

1. Citation update: The reference to Bandura will be updated to include the publication year (e.g., "Bandura, 2001").
2. Figure improvements: Figures 4–9 will be resized to ensure all text (axis labels, legends, annotations) is clearly legible in the final version.

Response to Review 1:

R1.Q1: Font size in Table 2. Thank you for your helpful suggestion. We will adjust the font size and optimize the layout of Table 2 (e.g., using multiple rows for tasks if needed) to improve readability in the revised version.