TreeTop: Topology-Aware Fine-Tuning for LLM Conversation Tree Understanding

We thank Reviewer 5ohp for their thoughtful feedback and for recognizing both the novelty of our conversation tree encoding method and its potential benefits to multi-turn, multi-party conversations. We also appreciate the reviewer highlighting how our work reveals important limitations in existing LLMs' ability to understand conversation trees. We address each concern below:

Weakness 1:

Reviewer Comment: "I did not catch how you finetune LLMs with the TreeTop framework. It is better to clarify it clearly in the main content."

Response: Thank you for the suggestion. We agree that the primary details relating to fine-tuning should be in the main content. We have added the following sentences to Section 4.1:

"Each model was fine-tuned on 10k randomly-sampled tasks from each of the 12 "fine-tuning tasks" listed in Table 2, holding out the other 5 "unseen" tasks for out-of-distribution evaluation. We adopt the unified encoding presented in Section 3.1. We list hardware and hyperparameter details in Appendix F."

Weakness 2:

Reviewer Comment: "It seems that the baselines and related works are out-of-the-date. Are there any other more recently works of LLM on conversation tree?"

Response: While we appreciate this concern about comparisons, we wish to highlight that ours is one of the nascent works on providing LLMs with a conversation tree understanding. Most of the recent LLM+graph works focus on traditional graph tasks like edge classification and node classification (see for example Pan et al., 2024; Tsitsulin et al., 2024; Chen et al., 2024b).

For non-LLM related benchmarks, we have compared against the most recent applicable works on our datasets, including PHAROS for rumor detection (Nguyen et al., 2024) and DFE-GCN for controversy detection (Hua et al., 2023). We note that a key challenge for expanding over datasets comparisons is data availability – publicly available datasets with complete conversation tree structures are increasingly scarce, especially from platforms like Twitter, which now require paid API access. Our work represents one of the first focused efforts to enhance LLMs' capabilities on conversation tree understanding.

Weakness 3:

Reviewer Comment: "I wonder how the conversation tree benefits the downstream dialogue tasks, such as dialogue response generation."

Response: Thank you for this insightful suggestion about dialogue systems. It's important to note that social media conversations have fundamentally different structural characteristics from traditional two-person dialogues (as with a chatbot). On social media platforms, multiple conversational threads emerge simultaneously in response to the same post, creating complex tree topologies that carry important signals about the discussion's nature (e.g., controversy, rumors). TreeTop specifically aims to exploit these unique structural patterns.

For example, a controversial post might generate multiple divergent reply branches. Similarly, a hateful post may display specific tree patterns -- these structural signals are distinct from traditional dialogue flows. Section 4.2 demonstrates how TreeTop effectively leverages these topology-specific signals, achieving strong performance on tasks like controversy detection (84.6% accuracy) and rumor detection (87.1% accuracy).

Our current framework is specifically designed to capture the unique multi-threaded nature of social media conversations, where topology provides critical additional signals beyond content alone. Nonetheless, we agree that exploring TreeTop's application to traditional dialogue systems is an interesting direction for future work.

We hope our responses adequately address your concerns. If you find these clarifications satisfactory, particularly regarding the fine-tuning methodology and the recency of our baselines, we kindly request you to consider raising your score.

Thanks for the reviewer's response.

We believe that our work is novel. By-and-large, the studies that use LLMs for graphs have tackled tasks on undirected, cyclic, static graphs, with applications geared toward standard GNN benchmarks (Hu et al., 2020a). In a parallel line of work, language models have been used to improve GNN performance. We describe these related works in Section 2 and Appendix B. Extending this area of research, we focus on fine-tuning LLMs to solve learning tasks on conversation trees, which are directed, acyclic, and temporal graphs capable of representing a wide variety of complex human interaction sequences. Our work opens a new area of research for LLMs, as well as providing the power of LLMs to the field of conversation tree analysis (or to text-attributed graphs broadly).

We have presented experiments on multiple datasets where conversation trees were available, and compared with the most recent results on those datasets. However, we would be happy to provide more empirical evidence and comparisons if the reviewer can point us in the direction of additional relevant papers and datasets.

Reviewer Question: "How sensitive are the model outputs to prompt design, and is there a mechanism to automatically adjust prompts for different tasks?”

To ensure that TreeTop models can translate natural language about tree structure into reasoning capability, during fine-tuning, we used multiple (2-3) prompt variants for each of the 12 primitive tasks. These prompt variants are listed in Table 10. This has multiple effects:

1. It allows TreeTop models to be robust to variations in the prompt, allowing it to effectively answer quantitative questions even when they are phrased slightly differently.
2. Importantly, it allows TreeTop models to generalize to unseen primitive tasks, specifically the 5 unseen tasks (see Section 4), which require new prompts on which were not part of fine-tuning.
3. It allows TreeTop models to be profitably trained on downstream tasks with entirely new prompts that were not used during primitive task fine-tuning.

Automatic prompt generation for different tasks is out-of-scope for the TreeTop framework; we consider this to be in the scope of systems that deploy TreeTop for various downstreams. Due to the above techniques, which inject prompt robustness into TreeTop models, we expect such strategies to be effective.

Thank you for the worthwhile questions. We respond to each below. We request that you please reconsider our work in light of these responses. If they have sufficiently addressed your concerns, please consider raising your score. Thank you!

Reviewer Question: "How are the 17 structural tasks weighted during training? Is their importance dynamically adjusted based on downstream application requirements?”

As described in Section 3.2, we only use 12 of the structural tasks during fine-tuning (i.e. the training that occurs after general pretraining). Each of these tasks are weighted equally -- downstream tasks are not taken into account. We find that this maximizes TreeTop's generalizability to a wide range of downstream tasks.

We chose uniform weighting for simplicity and reproducibility, as exploring differential weighting schemes would introduce significant additional complexity without clear benefits. Our results show that models fine-tuned with these 12 equally-weighted TreeTop structural tasks, and then further fine-tuned on various downstream tasks, consistently outperform models fine-tuned only on the downstream tasks.

Reviewer Question: "Can TreeTop model the associations between multiple conversation trees, especially based on longitudinal data or same user scenarios?”

In principle, this is possible, however we did not experiment with any downstream tasks involving longitudinal phenomena. What we have shown is that models fine-tuned on TreeTop's primitive structural tasks are the top performers on downstream tasks involving single conversation (in some cases, SotA). Importantly, we also showed that TreeTop models also generalize to unseen primitive tasks such as wedge/interaction/conflict detection (Section 4.1, Table 2). Together, these results strongly imply that TreeTop models will generalize well to tasks with tree-structured data, including inputs with multiple trees. Out-of-the-box, TreeTop models trained on single-tree tasks could be used to output representations of all (partial) trees in the input, and then an aggregator model could be used to deliver a final inference. On the other hand, fine-tuning models to consider cross-dependencies between trees would require new primitive tasks that are promising areas for future work. We have added the following sentence to our discussion section:

"Aside from multi-modality, another interesting future direction would be designing primitive tasks defined over multiple trees, involving resolving same-user behavior and cross-dependencies, as a way to fine-tune models for more complex longitudinal effects."

Reviewer Question: "How does TreeTop differentiate between structure- and content-related inferences when solving a reasoning task?”

Thank you for the interesting question. Please note that the structure and the content are strongly dependent -- for instance, a negative reply to the top post (depth 2) should be interpreted differently than a negative reply to the reply (depth 3). We discuss this further in the paragraph at L82. Due to this dependence, our TreeTop LLM prompt takes both the structure of the conversation tree and the content on the nodes of the tree as input, and TreepTop models are fine-tuned to draw inferences from both simultaneously. The powerful transformer-based architectures of LLMs are well-suited to learn which input features to focus on, given the rest of the input.

Reviewer Question: "How does TreeTop compare to other fine-tuning based methods on general graph-based reasoning tasks, such as edge existence prediction or node classification?”

The set of 17 TreeTop primitive tasks actually includes both edge existence and node classification for tree graphs. Our edge existence task is called "are_one_hop_neighbors". We also include meta-edge existence tasks such as "are_two_hop_neighbors", "are_three_hop_neighbors", "within_subtree", and "in_triangle". The set of tasks also include multiple node classification tasks including "depth", "is_leaf", "num_children", and "num_leaf_nodes". All these tasks are detailed in Appendix C.1. Note that TreeTop tasks are defined on temporal, acyclic, directed graphs, and are meant to be analogous to those defined on undirected graphs such as those found in e.g. Guo et al 2023 and other related works discussed in Section 2.

We sincerely thank Reviewer 8zwd for their time spent reviewing our paper. We also thank them for recognizing the novelty of our work in tackling challenges related to understanding conversation trees. We address the concerns raised as follows:

Weakness 1:

Reviewer Comment: "Some tasks are too easy to be meaningful. This work is proposed as a benchmark for LLMs and what's the point if a LLM can already achieve 100% accuracy?"

We appreciate the fair question, however, our experiments actually show that LLMs cannot achieve 100% accuracy on any task from the TreeTop framework without directly fine-tuning on that task. For instance, as shown in Table 2, Gemini's zero-shot and few-shot accuracies on the 12 TreeTop tasks chosen for fine-tuning largely fall in the 40%-70% range. Only after fine-tuning on those 12 tasks do accuracies increase to 93%-100%. Moreover, post-fine-tuning accuracy on the 5 tasks held-out during fine-tuning only increases to 63%-86%, showing that these tasks are in fact quite non-trivial. These results validate the core ideas of TreeTop: out-of-the-box, LLMs are not good at reasoning over structured data (see also Table 1). So, we introduce a suite of general, downstream-agnostic reasoning tasks on which they can be profitably fine-tuned (note that we do not propose this work as a benchmark). We show that fine-tuning on these tasks greatly increases the potential of models for out-of-distribution ("unseen") tasks and downstream applications.

Weakness 2:

Reviewer Comment: "Most datasets used in this study are derived from Reddit, which may limit the generalizability of TreeTop's effectiveness on other social media platforms or discussion types."

We respectfully disagree with the concern about dataset generalizability, as our experimental results demonstrate TreeTop's robust cross-platform effectiveness. The framework captures invariant structural properties of conversation trees that are fundamentally consistent across social media platforms. This is validated by our results on the PHEME9 Twitter dataset, where TreeTop-enhanced models achieve 87.1% accuracy - significantly outperforming prior approaches like PHAROS (75.9%) and NRA MOS-GAT (78.4%). The platform-agnostic nature of TreeTop is supported by our analysis of tree topologies, which shows that the framework learns structural patterns rather than platform-specific features. While it is true that most of our datasets come from Reddit, this is largely due to practical constraints: publicly available datasets with complete post-comment trees are scarce, especially from platforms like Twitter, which now require paid API access. Despite these data availability challenges, our transfer learning experiments demonstrate that models trained on Reddit data maintain strong performance on Twitter data without platform-specific adjustments (as shown by our results on the Pheme9 dataset). This effectiveness stems from TreeTop's focus on fundamental conversation tree properties that are consistent across platforms: hierarchical parent-child relationships, branching reply structures, and temporal dynamics. The topology of these trees follows measurable patterns in depth distribution and interaction dynamics across Reddit and Twitter. The empirical results establish that the TreeTop framework generalizes effectively across different social media platforms.

Weakness 3:

Reviewer Comment: "The paper briefly touches on potential issues such as over-moderation or promoting echo chambers but does not provide a detailed discussion on these risks or propose safeguards." We appreciate the opportunity to provide more detail. We have added the following sentences to Section 5. These sentences replace the sentence starting at L502 ("Second, we note that...") and continue until the end of the paragraph: "We also note that models such as those fine-tuned on TreeTop are often used for content-moderation (Roberts, 2017; Gillespie, 2020). A serious risk for such models centers around their potential misuse for content over-moderation, thereby promoting echo-chambers and insularity of thought (Kumar et al., 2024). Practitioners and platform designers should ensure that auto-moderation models trained with TreeTop (or any similar framework) are deployed responsibly, such as preferring “soft” moderation tags (shown in Martel & Rand (2023) to be effective) over bans/takedowns, providing definitional clarity to moderation policies, and offering transparency into moderation decisions, such as open-sourcing logs (Macdonald & Vaughan, 2024; Singhal et al., 2023). Additionally, our experiments show that TreeTop models are performant at early-detection (68.9% accuracy at 4 hours), which can enable effective human-in-the-loop moderation systems to be designed (Lai et al., 2022)."

Weakness 4:

Reviewer Comment: "The paper only includes results from one model (Gemini), which is bluntly telling the authors work for a certain company. Also not including other models fail to set meaningful baselines."

We appreciate the reviewer’s feedback, and yet would like to point towards the extensive experiments done using different classes of models – Gemini, PaLM-Bison and Gemma, while also considering ablation studies with Phi Mini and Mistral 7B (See Table 1, Page 6). The results show consistent improvement across all families - for instance, on structural tasks, Gemma-TT achieves a significant improvement in accuracy (90-100%) on most fine-tuning tasks and significant gains on unseen tasks (e.g., 80.8% on user interaction detection as compared with 22.1% accuracy prior to fine-tuning). Similarly, PaLM-TT shows strong results with 90-99% accuracy across most structural fine-tuning tasks on the hold-out set, validating that TreeTop's benefits generalize across model architectures.

For baselines, we provide extensive comparisons with other comparable approaches. For example, we compare against both traditional GNN architectures (GCN, GAT, GraphSage) and specialized models like DFE-GCN for controversy detection (76.6% accuracy vs TreeTop's 84.6%), PHAROS for rumor detection (75.9% vs 87.1%), and domain-specific approaches for winning argument detection. These comparisons demonstrate TreeTop's effectiveness is not model-specific but rather stems from its fundamental approach to structural learning.

Additionally, we validate our results through multiple ablation studies across models and datasets (detailed in Appendices G and I), ensuring that our findings generalize across different conditions. These empirical findings provide strong evidence for our framework’s general applicability across baselines.

Weakness 5:

Reviewer Comment: “Format: This paper uses a wrong font.” We thank the reviewer for pointing this out. While we were indeed using the ‘iclr2025_conference’ template available here, our inclusion of package ‘lmodern’ did conflict with a font. We have removed this offending package and uploaded a revision consistent with the styling requirements of ICLR.

We thank Reviewer 8zwd for their valuable feedback. We hope our comments and the revisions satisfactorily address your concerns. We request the reviewer to reconsider our work in light of these responses. If they think we have sufficiently addressed their concerns, we kindly request them to consider raising their score.

We thank Reviewer jnJR for their thorough and constructive feedback. We particularly appreciate their recognition of TreeTop's key contributions: the novel tree structure encoding method, the carefully designed structural tasks, and the framework's ability to generalize to unseen tasks. We are also grateful for acknowledging the comprehensiveness of our experimental validation through extensive ablation studies.

Reviewer Question: "For the primitive structural tasks, why is TT not compared against the baseline where the LLM is fine-tuned with non TreeTop encoded conversations and tasks? Please point it out if I understand it incorrectly."

Thank you for this important question about baselines. We have also conducted ablation studies comparing TreeTop against models fine-tuned on raw conversations without our structural encoding. Such an encoding captures the content of the conversation tree, without including the structure information about parent nodes and children (referred to as the PC encoding below). We show that TreeTop outperforms Gemini tuned on non-TreeTop encoded conversations and trees, thereby strengthening the need for our TreeTop framework. We have added this encoding ablation results in Appendix H. We include a sample result below to show this improvement (ZS refers to zero-shot and FT refers to fine-tuned; all numbers are f1 scores).

Controversial Post Detection

Fake News Detection

Rumor Detection

We thank the reviewer for raising the point on encoding ablations, which has helped make our experimental validation more comprehensive. We hope you find this clarification, and our additional experiments, useful towards acceptance of our work.