From Indicators to Insights: Diversity-Optimized for Medical Series-Text Decoding via LLMs

We sincerely thank the reviewer Cvyn for carefully reading our manuscript and providing thoughtful evaluations and suggestions.

Reviewer Concern (W1 & Q2):

W1: The proposed method relies on external tools to extract clinical indicators (e.g., RR intervals, QRS durations) for each input sequence, which may introduce computational overhead and hinder real-time deployment. The paper does not report the cost of computing these indicators.

Q2: In Section 5.5, could the authors clarify the efficiency evaluation setup, including batch sizes, memory usage, and whether the reported runtime accounts for indicator extraction? Since these indicators would need to be computed online in practical scenarios, please also specify the tools used and their computational overhead.

Response:
We have performed additional profiling experiments to quantify the indicator extraction cost and clarify the efficiency setup:

1. Indicator computation is extremely lightweight.
   We use widely adopted open-source signal processing libraries (YASA for EEG, NeuroKit2 for ECG). These operations are simple peak detection and morphology-based rules with $O(L)$ complexity.

   • On Sleep-EDF (EEG, 30 s), spindle and slow-wave detection takes ~3.4 ms/sample.
   • On PTB-XL (12-lead ECG, 5 s), extracting RR/PR/QRS intervals takes ~2.1 ms/sample.

2. Overhead is comparable or lower than other prompt-based methods.
   We benchmarked InDiGO against Time-LLM [12] with indicator or prompt generation included.

   Model	Indicator/Prompt Time (per sample)	1 Epoch Train Time (Sleep-EDF-20 / PTB-XL)
   InDiGO	3.4 ms (EEG) / 2.1 ms (ECG)	66.2 s / 788.8  s
   Time-LLM	2 ms (EEG) / 9ms (ECG)	156.3 s / 1742.1 s

Even when accounting for online extraction, the overall training and inference overhead remains comparable to or lower than other prompt-based methods, and does not constitute a bottleneck under our batch size (128) and memory configuration (~27 GB for GPT-2 XL).

3. Clinical indicators yield substantial gains that justify their small cost.
   Ablation (Figure 4) shows removing indicator-guided prompts leads to an average 1.6–3.1% Acc drop. In medical tasks where interpretability and accuracy are critical, this small computational overhead is a worthwhile trade-off.

Reviewer Concern (W2 & Q1):

W2: The proposed prompt optimization strategy is based on randomly masking parts of the initial prompt. It is unclear why masking informative parts of the prompt would lead to better performance, as complete prompts presumably contain richer semantics. Compared to this, prior work such as [1] employs LLMs to iteratively generate semantically different prompts, which may offer more principled and interpretable variations.

Q1: The authors should provide further explanation for why randomly masking tokens in the prompt leads to better model performance. What exactly is meant by “inaccurate prompts” in this context, and why are masked variants considered more “accurate”? A clearer justification, both empirically and conceptually, is needed.

Response:
We agree that the rationale behind our masking-based prompt optimization needs further clarification.

1. Motivation and “inaccurate prompts”.
   Our overarching goal is to provide a knowledge-enriched series–text joint decoding framework. However, high-quality series–text paired datasets with rich clinical descriptions are extremely scarce. As in many recent works, we must manually or semi-automatically inject domain knowledge into prompts. This paradigm inevitably introduces inaccurate prompts, which we define in two ways:

   • Semantic mismatch: Textual content such as generic dataset statistics (Time-LLM [12]), variable-relationship hints (KEDGN [22]), or timestamp descriptors (AutoTimes [21]) often lacks direct relevance to the current sample in medical classification tasks, leading to prompts that are textually correct but semantically unaligned.
   • Detection noise: Methods that rely on semi-automated tools for indicator extraction, including our approach and several recent works (e.g., Time-LLM [12], AutoTimes [21], PromptCast[40], ), obtain domain cues via external rule-based or automated detectors (e.g., RR/QRS intervals or sleep spindles). These tools are imperfect and may flag a waveform that does not actually exist in the current segment. Such erroneous cues can actively mislead the joint decoder.

   Under both conditions, “complete” prompts are not necessarily optimal; they may contain noise or irrelevant content that harms alignment between series and text.

2. Why masking improves performance.
   Our masking strategy is not intended to randomly discard useful information, but to expose the model to multiple variants of the same prompt under joint training with the downstream task objective. Combined with our Match Alignment (MA) and Diversity Optimization (DO), this achieves following effects:

   • By sampling different masked variants and jointly optimizing with the series–text interaction module, the model learns to focus on semantically consistent regions between the series and text.
   • Variants where masking removes noisy or misleading tokens (e.g., falsely detected waveforms) tend to yield higher task likelihood, and the model adapts to give higher weight to these “clean” regions.
   • Conversely, masking genuinely informative tokens does not improve performance, so their contribution is preserved during optimization.

   Empirically, this results in higher robustness to imperfect prompts and better generalization when prompts are partially incorrect—a common situation in medical deployments.

3. On iterative prompt generation vs. masking.
   We agree that works such as [1] (Guo et al. (2024)) explore LLM-driven iterative prompt generation. However, in medical series–text decoding this is fundamentally challenging: generating diverse, semantically rich prompts requires an LLM already capable of understanding the time series. Training such a cross-modal LLM in turn requires large-scale, high-quality series–text pairs—a resource that is currently unavailable in our domain. For practical medical classification tasks, the masking+MA+DO design provides a lightweight yet effective solution to the “imperfect prompt” problem without depending on unattainable data scale.

4. Empirical support.
   Table 3 and Figure 4-5 show that without masking, the model suffers a 2–4% drop on Sleep-EDF and PTB-XL. More importantly, prompts with partial masking consistently outperform “complete” but noisy prompts, demonstrating that our framework successfully filters out misleading textual content and enhances series–prompt alignment.

In summary, the masking strategy is not a substitute for richer semantics but a robustness mechanism against unavoidable prompt imperfections in knowledge-injection paradigms for medical time series. This design choice is both empirically validated and conceptually aligned with the realities of scarce, noisy series–text data in clinical domains.

Reviewer Concern (W3 & Q3):

Several key experimental details remain insufficiently reported, making it difficult to fully assess the method’s efficiency and fairness. See the Questions section for specific concerns.

It would be helpful to report statistics on the label distribution for each dataset. In medical classification tasks, label imbalance is common and could significantly affect performance metrics. In such cases, metrics like AUC might be more appropriate than accuracy.

Response:

1. Label distribution statistics.
   We have added the class distribution for each dataset:

   • Sleep-EDF-20: Wake 19.58 %, N1 6.63 %, N2 42.07 %, N3 13.48 %, REM 18.24 %.
   • Sleep-EDF-78: Wake 33.74 %, N1 11.01 %, N2 35.37 %, N3 6.67 %, REM 13.22 %.
   • PTB-XL: Arrhythmia 75.75 %, Normal 24.25 %.
   • HAR: 16.83 %, 15.99 %, 14.25 %, 16.66 %, 18.05 %, 18.22 %.

   We agree that label imbalance is common in medical classification tasks, which is precisely why our main results include Balanced Accuracy and Weighted F1, both of which explicitly account for class distribution.

2. Evaluation protocol and metrics.
   Our work follows established experimental setups and evaluation metrics from prior studies for each dataset:

   • Sleep staging: Brant-X [43], TinySleepNet [34], Brant -2.
   • PTB-XL: BIOT [41], NeuroLM [22].
   • HAR: Official benchmark splits.

   These works also adopt accuracy, Balanced Accuracy, and macro/weighted F1 as primary metrics. Following this convention ensures comparability with existing literature.

3. On AUC and additional metrics.
   We appreciate the suggestion to include AUC-based metrics. For PTB-XL, we have already reported AUCPR and AUROC in Table 2. For the sleep datasets and HAR, AUC is less commonly used due to the multi-class nature, but we are willing to include per-class ROC/AUC curves and micro/macro AUC scores in the camera-ready version to further strengthen the evaluation.

In summary, we have now reported class distributions, clarified that our metrics account for imbalance, and will incorporate additional AUC-based evaluations in the final version as suggested.

We thank the Reviewer U1ZG for the careful reading and constructive feedback.

W1: Although using masked Monte Carlo sampling can reduce the existing bias in manually designed prompts, the results are still extremely dependent on the expert-curated prompt prototypes. Nonetheless, the authors do not present an analysis regarding the relationship between these prototypes and InDiGO's performance.

We acknowledge that InDiGO leverages expert-curated prompt prototypes as entry points for domain knowledge. However, our goal is not to exhaustively enumerate all possible task-relevant indicators, but to demonstrate that even a small set of known, valuable indicators can be rapidly transformed into effective prompts—a non-trivial step in medical time-series analysis.

Indeed, our current experiments do not cover all potential indicators: for example, K-complexes and eye movements in sleep staging or QT/PP intervals in arrhythmia detection. In such cases, the proposed Match Alignment (MA) and Diversity Optimization (DO) mechanisms play a critical role in compensating for missing or imperfect prototypes by dynamically aligning and enriching the prompt distribution.

Importantly, this aspect has been explicitly analyzed in the main paper. In Figure 4 and Section 5.7, we investigated how InDiGO responds to progressively richer prototypes, ranging from minimal task instructions to detailed waveform relations, and compared these against strong baselines. This analysis was designed to ensure our evaluation of prototype sensitivity is systematic and complete. The summarized results are as follows:

These results demonstrate three key findings:

(1) InDiGO achieves significant gains over baselines even with only minimal task instructions, showing that it does not require an exhaustive set of handcrafted indicators to deliver strong performance.

(2) Performance scales with richer domain cues, confirming the effectiveness of indicator-guided prompts and their ability to inject clinical semantics into the model.

(3) The consistent gap between the full model and the version without MA/DO shows that these modules are essential in mitigating prototype dependency, dynamically enriching the prompt distribution and ensuring robustness under imperfect or sparse indicators.

Beyond these points, we would like to emphasize that this experiment also reflects a practical design philosophy: in real clinical workflows, obtaining a comprehensive set of perfect indicators is rarely feasible due to noisy semi-automated extraction and heterogeneous protocols. By showing that InDiGO performs well even with a single, partially informative prototype and automatically compensates for missing cues, we argue that our approach strikes a balance between robustness, generalizability, and real-world applicability. Furthermore, the architecture is fully compatible with multiple prototypes if domain experts can curate them, and we expect this to yield additional improvements in specialized settings, which we plan to explore in future work.

W2: While the masked Monte Carlo sampling reduces bias (shown contributions mathematically and empirically), it is not clear to me why the authors have not utilised multiple prototype candidates for a single prototype, such that the bias would be even smaller.

We thank the reviewer for the constructive suggestion. We agree that designing multiple prototype candidates for each task could further reduce prompt bias, and we initially considered this option. However, we observed two major limitations in practice.

(1) Reduced generality across tasks: In medical time-series analysis, a single framework must often cover diverse tasks (e.g., sleep staging, arrhythmia detection, wearable activity recognition). Maintaining multiple prototype sets per task would compromise portability and require manual adjustment when transferring to new domains.

(2) Increased error accumulation: Multiple prototypes also amplify the error rate introduced by automated indicator extraction tools, whose accuracy is inherently imperfect in clinical settings.

In light of these trade-offs, we deliberately chose to work with a single prototype and rely on our masked Monte Carlo sampling combined with diversity optimization to approximate a richer prompt distribution at low cost, achieving strong cross-task robustness without extensive manual curation.

That said, for domain experts with reliable indicator pipelines, integrating multiple handcrafted prototypes into InDiGO is straightforward and we expect it would indeed yield additional gains, which we plan to explore as future work.

W3: Figure 2 is bloated and it is hard to follow the flow and the steps depending on the figure alone. Although the text helps clairfying it to a certain degree, the figure can be improved for easier and quicker understanding.

We appreciate the reviewer’s feedback on Figure 2. We acknowledge that the current version is visually dense and may be difficult to follow without additional context. In future revisions, we are happy to revise the figure by streamlining the flow, abstracting low-level details, and enhancing visual hierarchy to improve readability. We believe these changes will make the framework easier to understand at a glance, even without relying on the text.

We thank the reviewer yNdq for the feedback and appreciate the opportunity to address several concerns, many of which appear to stem from a misunderstanding of our formulation and experimental setup.

Reviewer Concern (W1 & Q2):

W1: The assumption that the class-label probabilities follow a normal distribution in equation (1) does not make any sense. Since the rest of the method follows from this assumption, I am concerned about the theoretical validity of the proposed algorithm

Q2: To what extent does the performance of InDiGO depend on the normality assumption in equation (1)? Can this be modified or relaxed?

Response:

Our assumption in Eq. (1) does not imply that the true class labels are normally distributed. Instead, it follows a common formulation in probabilistic deep learning and uncertainty quantification for classification tasks, where the model’s predictive distribution over class logits is modeled as a Gaussian around the estimated mean \mu_i(s_i,t_i;\theta)\) with variance $\sigma^2_i(s_i,t_i;\theta)$. This is a standard surrogate likelihood for discrete outputs in the context of differentiable optimization, used in time-series modeling and Bayesian deep classifiers (e.g., Gal & Ghahramani, 2016 [1]; Kendall & Gal, 2017 [2]).

Importantly, the subsequent derivations in Sections 3–4 do not rely on a strict normality assumption. The key requirement is that the predictive distribution $P_{\text{LLM}}(Y|s,t)$ can be parameterized and integrated over the prompt distribution $P(t|s)$. Eq. (1) simply adopts an isotropic Gaussian as a tractable approximation to express that the network’s prediction is a distributional output dependent on both data and learnable parameters, rather than a point estimate. Other distributions (e.g., Dirichlet or softmax-based categorical likelihoods) could be substituted without affecting the core importance sampling and match-alignment objectives.

To further address Q2: We conducted a sensitivity check by replacing the Gaussian with a temperature-scaled softmax categorical distribution and observed <1% variation in key evaluation metrics across Sleep-EDF-20 and PTB-XL, indicating that InDiGO’s performance is largely agnostic to the exact parametric form of Eq. (1). This supports our claim that the framework’s effectiveness stems from indicator-guided prompt optimization and not from a strong normality prior. The results are summarized below:

Reviewer Concern (W2 & Q1):

W2: Additionally, the authors spend a lot of space in Section 4 detailed how the importance sampling method is derived; however, they do not provide sufficient context for how their method compares to previous works in the literature. As such, I have no sense at all of the methodological innovation of InDiGO over existing methods.

Q1: How does your InDiGO method compare methodologically to existing methods for integrating time series data and LLM's in the literature?

Response:
We agree that Section 4 focused heavily on the derivation of the masked Monte Carlo importance sampling (MCIS) mechanism, while the broader methodological context may not have been sufficiently emphasized. Below we clarify how InDiGO differs from and advances beyond existing series–text methods:

• Existing series–text works rely on static or single-point prompts.
  Prior approaches such as Time-LLM [12] and AutoTimes [21] concatenate a fixed prompt template with the series input, approximating $P(y|s,t)$ with a single handcrafted $t$. TEST [33] and KEDGN [22] attempt prompt alignment, but still depend on pre-defined prototypes without modeling the underlying distribution $P(t|s)$.

• No prior work explicitly approximates the marginal likelihood over prompts.
  Current methods do not consider $M = \int P(y|s,t)P(t|s)dt$ in their formulation. InDiGO is, to our knowledge, the first to treat prompt generation as a stochastic process and to approximate the marginal decoding objective via importance sampling rather than deterministic concatenation.

• Our MCIS is not generic importance sampling.
  InDiGO leverages indicator-informed proposal distributions $q(t|t_0)$ and combines masked perturbations with match-alignment and diversity optimization to dynamically evolve $q(t|t_0)$ during training. This differs fundamentally from naive prompt ensembling or static uncertainty sampling: the proposal distribution itself becomes task-aware through series–text co-attention and indicator-guided cues.

• Integration with domain indicators.
  Existing works either use dataset-level statistics (Time-LLM) or generic textual descriptions (KEDGN), which lack discriminative power in clinical tasks. InDiGO is the first to embed semi-automatically extracted physiological indicators into the proposal distribution, ensuring that the sampling space is grounded in domain knowledge.

To further make this distinction clear, we added a control experiment replacing MCIS with naive prompt ensembling (uniform random perturbations without indicator-informed $q(t|t_0)$):

Performance consistently degrades by 3–5% across key evaluation metrics when indicator-informed MCIS is replaced with naive ensembling, demonstrating that the gains of InDiGO stem not merely from sampling, but from the indicator-guided, dynamically optimized proposal mechanism.

We will revise Section 4 to explicitly connect these methodological differences with the existing literature to better highlight the innovation of InDiGO.

Reviewer Concern (W3):

W3: Additionally, the experimentation section is unclear and poorly written. It is not apparent at all from the main body of the text what the specific tasks on which the proposed method and its competitors are evaluated. While additional details are given in the appendix, as the paper is currently written it is difficult to parse the numerical results and evaluate them critically.

Response:
We respectfully disagree with the reviewer’s assessment and would like to clarify that the main body already provides explicit descriptions of the evaluation tasks at multiple points: 

• Section 5.1 (Datasets and Data Processing):
  We clearly state the mapping between datasets and tasks:
  • Sleep-EDF-20/78 — 5-class sleep staging under AASM standards using EEG/EOG signals.
  • PTB-XL — binary arrhythmia phenotype detection from 12-lead ECG.
  • UCI HAR — 6-class wearable motion recognition from accelerometer/gyroscope data.
  The modalities, sampling rates, and label spaces for each dataset are specified in this section. 
• Section 5.3 (Main Results):
  Before presenting Tables 1 and 2, we explicitly link each dataset to its corresponding task to make the reported metrics interpretable in context. For example, Table 1 is introduced as “5-fold cross-validated average results for sleep stage classification,” and Table 2 as “average performance on arrhythmia detection and human activity recognition.” 
• Section 5.4–5.7 (Ablations, Few-shot, Prompt Robustness):
  All ablations and low-resource evaluations are performed on the same clearly defined tasks, and we explicitly name the datasets and task settings (e.g., “Macro-F1 on sleep staging,” “Balanced Accuracy on arrhythmia detection”) in each subsection. 
• Appendix A (Additional Dataset Details):
  Provides segmentation strategy, subject splits, and preprocessing parameters to complement the main text.  This structure follows the same level of description as representative works in the domain such as BIOT [41] and Brant-X [43], which adopt the same datasets and provide comparable task descriptions in their main sections. We therefore believe the experimental setup is already presented with sufficient clarity for interpreting and critically evaluating the results directly from the main text. 

[1] Gal et al. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. ICML.

[2] Kendall et al. (2017). What uncertainties do we need in bayesian deep learning for computer vision? NeurIPS.

We would like to sincerely thank Reviewer cguB for providing a detailed review and insightful suggestions.

W1: Quality - Indicators must be hand specified per task, limiting out of the box generalization; authors themselves note dependence on structured indicator extraction.

Our approach indeed relies on domain-informed indicators, but we would like to clarify that these are not task-specific handcrafted rules; rather, they are essentially modality-level physiological primitives that are widely shared across applications within the same modality.

For instance, in EEG-based analysis, spindles and slow waves are not unique to sleep staging—they are also key features in seizure detection and anesthesia depth monitoring. Similarly, for ECG signals, RR intervals, PR intervals, and QRS durations are universally used in arrhythmia detection, QT interval abnormality screening, and heart failure risk assessment. Because these indicators reflect underlying physiological structures (e.g., PQRST complexes for ECG, characteristic EEG waveforms), the same extraction pipeline can be reused without major redesign when moving across tasks or datasets within a modality.

We agree that some dependence on structured extraction remains, and we acknowledge this as a current limitation. However, the modality-level nature of these indicators provides a degree of “out-of-the-box” generalization while maintaining clinical interpretability, which we see as an important trade-off for physiological and medical analysis.

W2: Complexity - Pipeline stitches multiple pretrained blocks and requires repeated prompt sampling; compute overhead vs. simpler fusion models is not fully quantified.

Our design combines a pre-trained series encoder and LLM to address the prompt–signal mismatch observed in medical tasks. The masked Monte Carlo sampling is included to reduce prompt bias rather than to add complexity; ablation results show that removing MCIS causes the largest performance drop (Table 3).

Regarding overhead, Table 4 provides quantitative comparisons: despite using multiple pretrained components, InDiGO(m=10) achieves lower inference latency than TimeLLM and similar training cost to KEDGN, while offering significantly higher accuracy. This suggests the additional sampling yields a favorable performance–efficiency trade-off.

We acknowledge the reviewer’s point that direct comparisons to simpler fusion models would further quantify the performance–efficiency trade-off. Our current study primarily focuses on domain-informed prompting and thus does not include plain series–text concatenation or vanilla feature fusion baselines. While such architectures may be computationally lighter, they lack mechanisms to integrate medical knowledge and, as evidenced by TF-C[44] and BIOT[41], do not achieve competitive performance on clinically grounded tasks, which is the core motivation behind InDiGO’s design.

We acknowledge the importance of simplifying the pipeline and are exploring adaptive sampling and model distillation to further reduce compute while retaining robustness.

Q1: What concrete criteria did you consider for choosing indicators per task? Were any indicators left out during part of the development process?

We based indicator selection on three concrete criteria: (1) physiological relevance, i.e., well-established links between the indicator and the underlying clinical state (e.g., spindles and slow waves for sleep stages, RR/QRS intervals for cardiac rhythm); (2) cross-task generality, prioritizing indicators that appear across multiple tasks within the same modality; and (3) automated extractability, focusing on features that can be reliably obtained via semi-automated tools to avoid manual annotation bottlenecks.

During early development we experimented with a broader set of candidates (e.g., EEG α-band spectral power, ECG T-wave alternans). Some were left out because they were highly redundant with existing indicators or exhibited unstable extraction quality that risked introducing noise. We also note that our current experimental paradigm does not yet incorporate every potentially useful indicator; for instance, K-complexes, which are known to play an important role in sleep-stage classification, are not included in the present version. Our aim was not to exhaustively enumerate all possible features, but to identify a core, transferable subset that balances clinical interpretability with robustness and practical extraction.

Q2 & W4: Necessity of Bidirectional Co-Attention and Impact of Simpler Cross‑Attention Variants

Our choice of bidirectional co-attention stems from the joint series–text decoding objective: both modalities carry complementary cues, and allowing each to query and update the other provides a stronger pathway for mutual alignment. As shown in Figure 5 of the paper, co-attention enables the series encoder to gradually focus on indicator-related prompt tokens, while the text branch refines attention toward task-relevant waveform patches.

We agree that a single cross-attention layer is more lightweight. We experimented with Perceiver-style setups using unidirectional cross-attention, with either series→text or text→series as the query. While both variants reduced computation, we observed a consistent performance gap compared to co-attention. For example, on Sleep-EDF-20 and PTB-XL:

This suggests that while cross-attention achieves reasonable alignment, having only one modality serve as the query weakens the contribution of the other and underutilizes complementary information. We see co-attention as a trade-off between computational overhead and robust bidirectional fusion, especially for clinical signals where modality synergy is critical.

Q3 & W4: No single‑prompt baseline (m=1) nor variant with simpler cross‑attention variants.

We conducted an additional experiment with m=1 to evaluate the impact of prompt diversity. The results are summarized below along with the w/o MCIS variant (no sampling). Compared to the full model (m=10), using a single prompt yields a noticeable drop, indicating that prompt diversity plays a critical role in reducing bias from any single textual description.

Q4: Does the GPT‑2 encoder half still use causal masks? If so, why is autoregressive masking desirable for encoding?

In our implementation the GPT 2 “encoder half” is used purely as a feature extractor, and we do not apply causal (autoregressive) masks during this stage. This allows the text encoder to attend bidirectionally over the entire prompt, which we found more appropriate for capturing the full contextual semantics of the indicator-guided descriptions.