Thank you for the encouraging review and the recognition of MAESTRO's performance and efficiency under practical settings of learning from heterogenous time-series modalities. We provide our response below :

It seems that the model involves multiple components that may require extensive hyperparameter tuning for the new datasets.

In our current work we have not extensively fine-tuned our hyper-parameters ( (as shown in Table 5 of the Appendix), yet we are able to outperform the existing baselines under complete and arbitrary modality combinations. However, for application-specific scenarios, based on the available computational resources, these hyperparameters can be selected to optimize for performance and computational efficiency. We present a brief sensitivity analyses for number of experts and the word length for the WESAD dataset across two computational platforms below :

Varying Experts for MoE (default w = 2)

Varying Word Length for SAX (default w = 2)

Thank you for commenting on this aspect. We will include this analyses in our Appendix.

MAESTRO is evaluated on classification tasks; discuss applicability to other tasks.

We currently explore classification tasks as they are the commonly occuring multimodal tasks (activity recognition, clinical disease identification, stress detection etc.) in consumer and clinical healthcare and lifestyle applications involving multiple heterogenous sensors collecting time-series data, which is the focus of our work. Also current multimodal methods with time-series are designed for predictive applications allowing us to benchmark MAESTRO.

We think, fundamentally MAESTRO's arechitecture can be extended to support generative tasks with mulitmodal input. In future work, we plan to explore two key directions for such extension: 1) curating a multimodal generative tasks dataset; and 2) exploring encoder-only vs. encoder-decoder cross-attention architectures for generation. One of the generative application spaces that stands to benefit from cross-modal attention to heterogeneous modalities without resource efficiency is robot manipulation. Recent works [1, 2] in robot manipulation explore enriching the latent representations of the action sequence through multimodal inputs before learning the Reinforcement-Learning policy. Often, there are time-series modalities [3, 4] that are availabile arbitrarily in these applications. We plan to explore the applicability of MAESTRO to such settings (to learn better action-sequence representation from multimodal inputs) in future.

We will add this discussion to our "Broader Impact and Future Work".

[1] Decision Transformer: Reinforcement Learning via Sequence Modeling, 2021

[2] A Vision-Language-Action Flow Model for General Robot Control, 2025

[3] MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation, 2024

[4] The MOTIF Hand: A Robotic Hand for Multimodal Observations with Thermal, Inertial, and Force Sensors, 2025

---

While MAESTRO handles missing data well, the model's behavior under noisy inputs is not systematically evaluated. I am curious about its robustness to the noisy data.

In the current work, we are focused on handling complete modality missingness, which is a common issue in real-world sensing applications with a large number of modalities (N > 4), where a sensor fails and we are left with an arbitrary set of multimodal inputs to make predictions, as mentioned in our paper's scope (lines 10, 62, 74).

While an extensive study on noise robustness is out of scope for this paper, which focuses on robustness to missing modalities, it could be a promising direction for future work on MAESTRO. We have just conducted some initial exploration with three types of noise defined as follows. Here $\tilde{\mathbf{x}}$ denotes the subset of modalities chosen for introducing corruption or omission :

1. Random noise : Selected channels are replaced with i.i.d. Gaussian noise, $\tilde{\mathbf{x}} = \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(0, \sigma^2), \quad \text{for } c \in \mathcal{C}$
2. Additive noise : Gaussian noise is added to the original signal, $\tilde{\mathbf{x}} = \mathbf{x} + \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(0, \sigma^2), \quad \text{for } c \in \mathcal{C}$.
3. Additive noise with spikes (representing electrical spikes) : Sparse high-magnitude impulses (spikes) are added in addition to Gaussian noise: $\tilde{\mathbf{x}} = \mathbf{x} + \boldsymbol{\epsilon} + \mathbf{M} \odot \mathbf{S}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(0, \sigma^2)$, where $\mathbf{M} \sim \text{Bernoulli}(p)$ is a binary mask indicating spike locations with $p$ probability, $\mathbf{S} \in \{-m, +m\}$ is the spike magnitude, $\odot$ denotes element-wise multiplication

We present the zero-shot performance of MAESTRO on noisy WESAD data in the following two combinations : in combination 1, we dropped the chest accelerometer, and in combination 2, we dropped the wrist BVP, EDA, and Temp. Please refer to Table 6 in our Appendix for more details on the WESAD modalities.

We observe that for simple noise settings like random noise, MAESTRO treats those modalities similar to complete missingness of that modality.

In the simple additive noise setting, we are able to perform almost as well as the case where all modalities are present. This is not surprising because (1) our SAX representation provides robustness to local perturbation, and (2) our cross-modal attention allows querying only for the relevant inter- and intra-modal values relevant to the task.

Finally, a more aggressive noise where we simulate electrical noise interference as spikes shows the same performance in one combination as missingness and in another combination very close to the full modality scenario.

We can consider missingness to be one of the aggressive noise corruptions, and through this controlled experiment, we can see MAESTRO's robustness to simple perturbations. Thank you for raising this aspect of MAESTRO's robustness to noisy inputs. We will include the above preliminary analyses in our Appendix to motivate and guide potential future exploration, e.g., to explore other systematic noise injection mechanisms and their impact on MAESTRO.

---

Minor writing issue: "reliance on" in line 5 should be put ahead of "(1)"?

We will update our abstract to reflect the suggested flow.

Thank you for your encouraging review and constructive feedback. We appreciate your recognition of the practical significance of our work and the merit in MAESTRO's design for real-world sensing applications. Please find our response to your questions below:

Could the authors clarify how the gating network 𝐺 is trained?

The gating network, $G$, is composed of two feed-forward layers, and the output is the sigmoid of the last layer which produces an $M$-dimensional logit vector (where $M$ is the number of modalities). We train this gating network using the cross-entropy loss (given in lines 280-281) with the end-to-end MAESTRO. We do not use any additional regularization loss. During training, we follow a curriculum learning approach (as mentioned in lines 253–257), which increases the modality dropout monotonically over the course of training epochs, as shown in Figure 9 of the Appendix. This training strategy helps prevent overfitting to any specific modality combinations.

Additionally, in our initial design iterations, we considered a loss term—sparsity loss—where we aimed to minimize the overall attention budget. The sparsity loss was constructed by taking the sum of the $M$-dimensional logit vector. The final loss considered was the cross-entropy loss and the sparsity loss. However, this did not result in stable training without constraining the sparsity loss. In future work, we will consider proactively sparsifying the attention budget based on the available computational resources.

Could the authors provide more quantitative or qualitative insights into why SAX representation improves performance (e.g., beyond Table 2 and Figure 3)? Would the model still benefit from SAX if trained on high-resolution sequences with sufficient compute?

The key advantages of discrete binning achieved using SAX specifically for MAESTRO are: 1) the ability to reserve a symbol to naturally encode missingness, and 2) to reduce sequence length.

An additional benefit of SAX—particularly for predictive tasks where discriminative patterns across the complete sequence inform task-specific predictions—is that mapping to a fixed vocabulary helps suppress local noise or spurious perturbations. This allows the model to focus on high-level trends that are more relevant to the task. Many previous works [1, 2, 3] have reaped this advantage and reported improved classification performance.

In our case, to demonstrate the benefit of SAX tokenization, we conduct following two experiments.

Experiment I: Using the WESAD dataset (stress detection with 5 modalities and 3 classes), we trained on high-resolution data without reducing sequence length by setting the word length (w) to 1. The results presented below confirm the benefit of SAX representation.

Experiment II: We add small Gaussian noise to 20% of the modalities (in this case, the wrist BVP, TEMP, and EDA) and conduct zero-shot inference. As expected, when the word length is set to 2, the effect of this small perturbation is neglible. Even with word length 1, the performance is much more robust compared to raw input

We will add these experiments to our Appendix section and refer to them in Section 3.2.1 to give our readers more intuition about SAX-based tokenization. Thank you for asking about this aspect.

We would also like to draw your attention to Figure 5 and Section E.1 in the Appendix, which demonstrate that under arbitrary missingness, the proposed MAESTRO with SAX encoding outperforms its counterpart without SAX encoding.

[1] Symbolic representation for time series, 2024

[2] Bake off redux: a review and experimental evaluation of recent time series classification algorithms, 2024

[3] SAX-VSM: Interpretable Time Series Classification Using SAX and Vector Space Model, 2013

Could the authors discuss how the current design generalizes to the settings of asynchronous modalities (e.g., with misaligned sampling rates and no temporal sync), and whether additional temporal alignment strategies would be needed?

In the current work, we focus on handling complete modality missingness, which is a common issue in real-world sensing applications with a large number of modalities (N > 4). This typically occurs when a sensor fails and we are left with an arbitrary subset of multimodal inputs for prediction, as described in our paper's scope (lines 10, 62, 74).

While a comprehensive study evaluating MAESTRO on irregularly sampled, asynchronous time-series is out of the scope of the current paper, we present a brief pilot study in a simplified setting. We mask approximately 25% of the samples from the beginning and end of the sequence in 20% of the modalities(in this case, the wrist BVP, TEMP, and EDA) during inference to demonstrate zero-shot transfer capability. Currently, the model treats this partial missingness similarly or slightly better than complete missingness.

This performance can potentially be improved by incorporating standard techniques such as absolute positional encoding, or employing prior models like mTAN [4] (multi-time attention network), which can better encode irregular time-series with semantic meaning to support downstream cross-modal learning. We will include this discussion and pilot experiment in our Appendix, and plan to explore these directions in the future. Thank you for commenting on this.

[4] Multi-Time Attention Networks for Irregularly Sampled Time Series, 2021

Could the authors discuss performance under strict compute budgets (e.g., edge devices)? How does inference latency scale with modality count and sequence length, especially in high-frequency sensor settings?

We conduct two experiments to demonstrate 1) MAESTRO's performance across two computation platforms and 2) benefit of sparse attention with simulated high sampling rate data stream.

Experiment I: We implement MAESTRO with different number of experts and varying word lengths (which in turn reduce the sequence length), and report the accuracy, GFLOPs, and latency on two compute platforms—RTX A6000 and Jetson TX2.

Varying Experts for MoE (default w = 2)

Varying Word Length for SAX (default w = 2)

Experiment II: We resample the modalities in WESAD to increase the sequence length in order to emulate high-frequency sensor readings and report the performance below. As expected, our proposed MAESTRO performs consistently even with longer sequence lengths. However, upon replacing the sparse attention with canonical dense self-attention layers in both the intra-modal and cross-modal stages, we encounter out-of-memory (OOM) issues at higher sampling rates (in this case, 128 Hz), highlighting MAESTRO’s advantage in resource efficiency. All experiments were conducted on the RTX A6000.

Thank you for the encouraging review and the constructive feedback on improving our manuscript. We provide our response below :

Provide inline explanation of the symbolic tokenization instead of citing original reference for better readability.

Thank you for the suggestion. We will add in the revision the following description of the breakpoint generation and mapping in the main text for easy referencing of the reader :

"To ensure an equiprobable distribution of symbols, we follow the empirical observation by Lin et. al [28] that normalized time series sequences follow a Gaussian distribution. We partition this distribution into $\alpha$ equal-sized areas under the Gaussian curve, referred to as regions, using breakpoints $\{\beta_0, \beta_1, \ldots, \beta_\alpha\}$, where each region corresponds to a unique symbol. The normalized piecewise aggregated value $\hat{x}[w]$ is then mapped to a symbolic token $s[w] \in \{s_1, \ldots, s_\alpha\}$ based on the breakpoint interval in which it falls, denoted using the mapping function $\phi(\hat{x}[w])$."

---

"max-mean sparsity metric" from Zhou et al [63] used to compute the sparse self attention seems like a key detail but it's delegated to appendix + citation.

Thank you. We will move the max-mean sparsity metric details from Appendix B to the Section 3.2.2 to improve readability.

---

a(m_i, theta_a) is introduced in line 195 in 3.2.2. but m isnt defined until the next section, which causes the reader extra mental burden in the earlier section.

We will reorganize our writing to define $\mathbf{m}_i$ in Section 3.2.2 and explicitly mention that the design of $a(\mathbf{m}_i; \theta_a)$ is detailed in the following Section 3.2.3. Here is the planned updated text from line 195 :

".... with a key modification: the sparsity budget is adaptively controlled by the learned function $a(\mathbf{m}_i; \theta_a)$, where $\mathbf{m}_i \in \mathbb{R}^{1 \times M}$ is a logit vector indicating the presence of each modality for an input sample $s^j_i$. Details on training $a(\mathbf{m}_i; \theta_a)$ are provided in Section 3.2.3."

---

Clarification of the experimental setup in the ablation studies -- Were they done with or without SAX? Difference between full vs. all-full attention.

We conduct two kinds of design deep-dives in Section 4.2 –

1. Ablation of the design components in Figure 6: The first group of bars shows the ablation without SAX and demonstrates that this tokenization leads to an 8.5% and 5% improvement under full and 40% missingness in modalities during inference, respectively. For the remaining components, we conduct experiments with SAX (as it is a part of MAESTRO unless explicitly mentioned otherwise).

2. Sparse attention-derived Computational efficiency of MAESTRO: All experiments in this section are conducted with SAX. In the first two rows after MAESTRO, Full-Attn (Per-Modal) and Full-Attn (Cross-Modal) refer to variants where only the per-modal or cross-modal components, respectively, are replaced with dense attention. All Full-Attention indicates that both the per-modal and cross-modal attention modules use dense attention mechanisms instead of the proposed sparse design. We will update in the revision the caption of Table 3 as follows :

   "Computational Complexity. In MAESTRO, sparse attention in the Per-Modal and Cross-Modal components is replaced by dense attention, referred to as Full-Attn (Per-Modal) and Full-Attn (Cross-Modal), respectively. Replacing all sparse attention components with dense attention is denoted as All Full-Attn."

Thank you for your encouraging feedback on our work's pragmatism, performance and robustness. We provide our response below :

Even though the results are good, the algorithm and techniques largely combine those of existing works.

We acknowledge that prior advancements in the fields of data mining [1] and long-horizon time-series forecasting [2] have inspired us; but our contribution lies in adeptly leveraging these ideas to solve a different problem--learning from dynamic and heterogeneous multimodal time series. We propose adapting the sparse attention technique at inter- and intra-modal stages with a learnable attention budget (the gating function in Section 3.2.3); using the opportunity with discrete symbols in tokenization to reserve a symbol for missingness (Section 3.2.1); and incorporating Mixture-of-Experts (MoE) to allow for implicit expert specialization under arbitrary modality combinations (Section 3.2.5)—which significantly extend beyond direct combinations of existing techniques. Our ablation study in Figure 6 shows the effectiveness of these proposed techniques.

Additionally, our work presents an alternate viewpoint of multimodal handling of multi-sensor data over the commonly adopted multivariate modeling, which is often chosen because the sensing modalities share the same (numerical) representation in the input space. Our design considers the heterogeneity in the sensors and proposes an effective multimodal learning framework, MAESTRO.

---

Justification of how the algorithm overcomes the many missingness patterns challenge.

We conduct an ablation of all our proposed components under complete observation and under modality missingness to demonstrate their roles (in Figure 6 of Section 4.2). The key intuition is that cross-modal attention (which improves performance by 21%) allows each modality to adequately query the necessary modalities—including itself—based on their availability and relevance. The modality dropout augmentation (improves performance by 9%) and the modality position embedding (improves performance by 16%), which denote the ordering of the modalities, help facilitate this learning. Supporting cross-modal learning with modality-specific embedding and augmentation helps MAESTRO achieve consistently superior performance (an average improvement of 9%) over other baselines, even under 40% missingness.

---

Minor: The algorithm involves several hyperparameters (e.g., window size $W$, $\alpha$) without theoretical guidance for tuning, adding training overhead.

Thank you for commenting on this aspect. In our current work we have not extensively fine-tuned our hyper-parameters (as shown in Table 5 of the Appendix), yet we are able to outperform the existing baselines under complete and arbitrary modality combinations. However, for application-specific scenarios, based on the available computational resources, these hyperparameters can be selected to optimize for performance and computational efficiency. We present a brief sensitivity analyses for number of experts and the word length for the WESAD dataset across two computational platforms below :

Varying Experts for MoE (default w = 2)

Varying Word Length for SAX (default w = 2)

We will include this analyses in our Appendix.

---

[1] Experiencing SAX: a Novel Symbolic Representation of Time Series, 2007 [2] Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, 2021