MuHBoost: Multi-Label Boosting For Practical Longitudinal Human Behavior Modeling

We thank Reviewer qBMd for your thorough review of our paper. We would like to respectfully present our rebuttal as follows.

Weaknesses

W1 - Needs more background clarification

We appreciate your concern regarding the background of SummaryBoost. To clarify, it was proposed to specifically address tabular data classification only with small sample sizes and high heterogeneity. Therefore, we do not include it in the background or related work, though we did clarify its relevance to ubiquitous health data in the Introduction (in Contribution I). For MLC approaches such as LP and CC methods, we introduced some background and discussed relevant works in Section 2 and Appendix B (under MLC subsection for both). We also clarified important works upon which our methods are built throughout Section 3 as well as Appendix D.1 (for MuHBoost[CC]).

W2 - Comparison to multilabel text classification

Existing methods in multilabel text classification (MLTC) mentioned in the Related Work (last paragraph in Appendix B) have only been applied to generic text labeling tasks on large datasets (with thousands of samples). Furthermore, to our best knowledge, none employ commercial LLMs such as GPT (mostly on pretrained transformers such as BERT instead).

W3 - Statistical significance

We ranked the considered methods in Table 1 by the average of the respective performance measures (Hamming accuracy, macro F1, and micro F1) across 10 different data splits. Therefore, some results are not statistically significant (i.e., ties, particularly among the baselines). However, we have double-checked using paired sample t-test with significance level of 0.05 and confirmed that (i) our three proposed methods strictly outperform all baselines, (ii) the LLM-based baselines (zero-shot and few-shot methods) strictly outperform other baselines.

W4 - Only binary class predictions

We believe binary class prediction aligns with most applications in longitudinal human behavior modeling, where the goal is often to diagnose certain health/well-being outcomes such as depression, stress, and problematic use of various drugs. Therefore, we only considered binary class prediction for all tasks in this work. Conceptually, because our methods extend SummaryBoost (which considered single-label binary or multiclass classification) to MLC, extending them to multiclass MLC (where some or all of the Q labels are multiclass) is straightforward, as mentioned earlier in Footnote 6. We will consider extending our methodology and experiments in future work.

Questions

1. How do the authors combine data descriptions across topics?

For clarifications, in the first step of our data conversion prompt, when the number of topics is reasonably small as illustrated in Figure S4b (newly added based on your review), we fit all topics within the data conversion prompt, hence there is only one data description for each record. When there are many topics, we split them into groups and respectively call the LLM to describe each group, resulting in multiple data descriptions. Then, in the second step, we append these data descriptions together (i.e., separated by newlines) to form a complete data description of the record.

2. For the two psychology datasets (LifeSnaps and GLOBEM), why are some tasks (e.g., negative emotions or anxiety) defined differently across datasets?

Following your review, we have clarified the factors deciding our label definitions for all considered datasets in Appendix C.1. For GLOBEM, we defined the label for PANAS task following the definition from (Englhardt et al. 2024), which is grounded on domain literature. For LifeSnaps, which studied participants in a different geographical region (Europe instead of U.S.), we simply defined the PANAS task for demonstration purposes only given the lack of conclusive findings regarding PANAS cutoffs in the area.

3. When describing baselines in Section 4.2, what do problem transformation and algorithm adaptation mean?

We would like to categorize the baselines given the two main approaches for MLC, which were first introduced in Section 2 (under MLC subsection).

Minor comments / suggestions

Throughout I recommend using parenthetical citations (citep instead of cite) Line 081: “have the two following” Line 206: “require” instead of “requires”

We have incorporated these helpful suggestions in our revised paper.

At multiple points, footnotes referenced in the main text were placed in the appendix. Could each footnote be placed on the page they are first referenced on?

Due to the page limit, we resort to placing some footnotes in the appendix (only when they are mentioned in both the main text and the appendix).

We thank Reviewer 8yZu for taking the time to review our paper. We would like to respectfully present our rebuttal as follows.

Weaknesses

Iterative improvements on prior SummaryBoost work with limited novelty

While we indeed based our approach on SummaryBoost, it is not immediately clear how a tabular data classification method for small datasets can be applied to our problem of TSC given ubiquitous health data. Because this general form of time-series data may have high heterogeneity in practice (which has yet to be addressed by prior works as discussed in Appendix A), we leverage SummaryBoost to circumvent this challenge given its efficacy in heterogeneous tabular data classification and employment of LLMs (proven to be suitable for longitudinal human behavior modeling). Thus, we believe the novelty of our work comes mainly from the approach, rather than the individual techniques.

The new data conversion uses a seemingly straightforward two-step approach based on LLM prompting in Section 3.1. The prompting itself does not seem to be very innovative, and while Table 2 does demonstrate an empirical performance increase, it is unclear how exactly each step of the two step approach helped.

Following up on our previous point, while the data conversion procedure is based on prompting techniques from existing works in various fields, it can effectively deal with general time series with high heterogeneity and dimensionality. Moreover, it can accommodate any auxiliary background data from the records, which can also be heterogeneous and high-dimensional (e.g., PWUD dataset). To our best knowledge, this flexibility is unseen in relevant works, which highlights the novel contribution of our data conversion approach.

Regarding the individual performance of our two-step procedure, because the second step requires the data description(s) in complete sentences as input, we can only ablate either (i) the second step or (ii) both steps altogether. Based on your review, we have extended our ablation in Appendix C.3, under Further Ablation. As shown in Table S11, ablating (ii) results in a sharp decline in predictive performance compared to ablating (i), which implies more impact from the first step.

MuHBoost modifies the original ClusterSampling algorithm by emphasizing rarer classes in the sampling procedure, then the follow-up methodologies MuHBoost[LP+] and MuHBoost[CC] seem to be straightforward adaptations of prior methods from Nam et al., 2017 and Adaboost.C2.

Although the two MuHBoost variants are based on existing works, we believe our work is the first to adapt their methodologies to LLMs. For (Nam et al., 2017), which reformulates MLTC problems and redesigns RNN architectures for generic text labeling, we adapted their idea by redesigning the inference directive in the inference prompt of LLMs, without the need for architectural design. Albeit straightforward, our approach effectively reduces the computational complexity of the problem and hence may alleviate hallucinations from LLMs as empirically shown in our newly added MuHBoost vs. MuHBoost[LP+] under Appendix C.3.

For (Li et al., 2023), we apply the idea of combining AdaBoost with classifier chains (using traditional classifiers i.e., SVM for building weak learners) to enable LLM-based boosting classifier chains, which is novel to our best knowledge. Because this transforms MLC problems into multiple binary classification problems, our adaptation brings an alternative way of employing LLMs for MLC without the need for prompting multiple related questions simultaneously, which risks hallucinations especially for complex prediction tasks. Overall, we respectfully stand by our view that despite being straightforward and built upon prior works, the proposed methods are still novel in the sense of how they approach our (nontrivial and understudied) problem of interest in a simple-yet-effective way.

Prior work section does not seem to contextualize the importance and relevance of SummaryBoost, especially in context of the longitudinal ubiquitous computing domain.

For clarification, SummaryBoost was proposed to specifically address tabular data classification with small sample sizes and high heterogeneity. Therefore, we do not include it in the related work as it cannot be applied to the longitudinal ubiquitous computing domain. We previously clarified its relevance to ubiquitous health data in the Introduction (in Contribution I). This unique approach to longitudinal human behavior modeling is hence the main novelty of our work.

Questions

\cite is incorrectly used throughout the paper when \citep should be used

We appreciate Reviewer 8yZu for noticing our inadvertent mistake. We have taken this into account in our revised paper.

Dear Reviewer 8yZu,

As the end of the discussion period approaches, we would like to check in with you on whether we have addressed your concerns of our paper. We are happy to discuss further if that is not the case.

Sincerely,

Authors of the current ICLR submission

Dear Reviewer 8yZu,

While we will not be able to upload our revised PDF after today (November 27 UTC), we can still address any of your further concerns. Please let us know if there is anything else we can provide to improve our paper.

Sincerely,

Authors of the current ICLR submission

We thank Reviewer aJj3 for your thorough review of our paper. We would like to respectfully present our rebuttal in the following.

Weaknesses

1. For clarification, in our paper, we refer "high-dimensional time series" to those having a large number of features recorded over time and "long time series" to those consisting of many time points (i.e., either collected at high frequency or over a long period of time). While we indeed only conducted experiments on datasets with at most 30 time points, our methods can generally deal with long time series via the data conversion procedure as clarified in Appendix D.4 following your review. Unfortunately, we are not able to extend our experiments to longer time series due to the limitations of currently available ubiquitous health datasets, and hence we acknowledge this limitation from our work and have mentioned it in Appendix D.6.

2. Although the two MuHBoost variants are based on the two stated existing works, we believe our work is the first to adapt their methodologies to LLMs. For (Nam et al., 2017), which reformulates MLTC problems and redesigns RNN architectures for generic text labeling, we adapted their idea by redesigning the inference directive in the inference prompt of LLMs, without the need for architectural design. Albeit straightforward, our approach effectively reduces the computational complexity of the problem and hence may alleviate hallucinations from LLMs as empirically shown in our newly added MuHBoost vs. MuHBoost[LP+] under Appendix C.3. For (Li et al., 2023), we apply the idea of combining AdaBoost with classifier chains (using traditional classifiers i.e., SVM for building weak learners) to enable LLM-based boosting classifier chains, which is novel to our best knowledge. Because this transforms MLC problems into multiple binary classification problems, our adaptation brings an alternative way of employing LLMs for MLC without the need for prompting multiple related questions simultaneously, which risks hallucinations, especially for complex prediction tasks. Overall, the novelty of our methods comes mainly from the approach rather than the individual techniques.

Questions

1. Recall that SummaryBoost was designed specifically for tabular data (Manikandan et al., 2023) and does not directly apply to longitudinal human behavior data. On the other hand, XGBoost can be applied to both forms of data. Therefore, it is impossible to compare SummaryBoost and XGBoost in our settings. However, following our results from Table 1, our methods still outperform XGBoost on LifeSnaps and GLOBEM, which contain mostly numerical features (in both time-series and auxiliary data). This can be attributed to the fact that while XGBoost works especially well for tabular data (Borisov et al., 2022), the same may not apply to longitudinal human behavior data, highlighting the contributions of our work.

2. As mentioned in Appendix C.2 when describing our baselines, for MLkNN and MLTSVM, we embedded the data descriptions via OpenAI’s embeddings API, which are then fed into these classifiers as input features. This is consistent with how the work of SummaryBoost (the foundation of our work) processed the data descriptions for their kNN baseline.

3. Following your suggestion, we have included another baseline based on RNN (Du et al., 2021), which was recommended by the authors of GLOBEM, and found that its performance is substantially worse than vanilla MuHBoost (and hence the two MuHBoost variants as well). Please refer to Additional Benchmarking in Appendix C.3 for further details.

References

(Borisov et al., 2022) Deep Neural Networks and Tabular Data: A Survey.

(Du et al., 2021) AdaRNN: Adaptive Learning and Forecasting for Time Series.

We thank Reviewer wDHz for taking the time to review our paper. We would like to first comment on the weaknesses and then address your questions.

Weaknesses

1. Recall that our main goal is to develop methodologies that mainly consider the three stated challenges of ubiquitous health data (and two shortcomings of state-of-the-art i.e., LLM-based approaches). These challenges include small sample size due to the typically high data acquisition cost. You are correct that this brings a limitation to our work. Based on your suggestions, we have included a Limitation section in Appendix D.6. More specifically, we do not expect our proposed methods to scale to larger datasets (i.e., with thousands of samples) in terms of resource efficiency. When the opportunity to work with large ubiquitous health data arises, approaches that require ample sample size such as LLM finetuning become more suitable than ours i.e., boosting LLM-generated weak learners, which is also the case for SummaryBoost.

2. Yes, you are correct. Following up on our previous point, our approach focuses on ubiquitous health datasets that typically come with small sample sizes. Therefore, we do not claim the scalability of our developed methods for large datasets. Nevertheless, we believe the proposed idea of employing LLMs for MLC in real-world health prediction tasks is novel (as discussed in related work from Appendix B) and could be applied to significantly reduce resource consumption for data-extensive approaches such as finetuning (which is prohibitively expensive when the number of health/well-being outcomes to predict, Q, is large).

3. We apologize for any confusion in our presentation. During writing, we tried to follow the outlines and technical details provided in the SummaryBoost paper from NeurIPS-23, which forms the foundation of our methodology. In the paper, the authors presented their work with only algorithms and illustrations, and hence we adopted this convention. We are willing to elaborate on our derivations if Reviewer wDHz could provide more details on which "principle formulas" are in question.

Questions

1. Based on your suggestions, we have further clarified how our proposed methods address the three challenges of ubiquitous health data compared to existing LLM-based approaches in Appendix D.4.

2. Our current literature review has already provided a thorough coverage on MLC, as we are not aware (as of writing this rebuttal) of any innovative work from established venues in recent years. For MLTC, we have included our updated literature review. Please refer to the last paragraph in Appendix B for a discussion on more recent works.

3. Based on your suggestion, we have provided more solid proof. In particular, in Table S10 (newly added from your review), we show that MuHBoost underperforms MuHBoost[LP+], particularly for PWUD dataset where the number of labels Q is larger. For further details, please refer to MuHBoost vs. MuHBoost[LP+] in Appendix C.3.

The hallucination of LLM is a widely and deeply studied topic, the authors haven’t provided clearly...

Our approach actively alleviates LLM hallucinations during (1) the data conversion procedure and (2) the inference process (via the two MuHBoost variants). For (1), we demonstrated its positive effect through our ablation study Impact of Refining Data Description in Section 4.3. To further support our claim based on your review, we have conducted another ablation study where we omit the entire data conversion procedure (Table S11). Please refer to Further Ablation in Appendix C.3. For (2), please refer to MuHBoost vs. MuHBoost[LP+] in the same appendix for how our two MuHBoost variants can mitigate hallucinations from LLMs.

For section 3.1: The data transformation process could be graphically illustrated...

Based on your suggestion, we have included an illustrative example in Figure S4 of our revised paper.

4. In existing longitudinal human behavior modeling studies, all state-of-the-art approaches were compared against these baselines. Therefore, we naturally considered them as baselines following the convention. We also compared our methods to those state-of-the-art approaches. Furthermore, based on your review, we have included another baseline (Du et al., 2021), which was recommended by the authors of GLOBEM as a potential solution to longitudinal human behavior modeling. Our results in Table S12 show that MuHBoost still outperforms this baseline. Please refer to Additional Benchmarking in Appendix C.3 for further details.

5. Based on your suggestions, we have included an analysis of time complexity in Appendix D.5. Because it may take arbitrary time to call LLMs (e.g., intermittent delays), we do not report empirical runtime in our experimental results (see also Footnote 16). We have also included a paragraph on FLOPs estimation in Appendix D.5.

Reference

(Du et al., 2021) AdaRNN: Adaptive Learning and Forecasting for Time Series.

Dear Reviewer wDHz,

As the end of the discussion period approaches, we would like to check in with you on whether we have addressed your concerns of our paper. We are happy to discuss further if that is not the case.

Sincerely,

Authors of the current ICLR submission

Dear Reviewer wDHz,

We are glad to hear that we have addressed your concerns. Again, we greatly appreciate your helpful review of our paper! Your comments and suggestions have significantly improved the overall quality of our work.

Sincerely,

Authors of the current ICLR submission