Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos

On multimodal pretraining requirements

Thank you for raising this question. We assume that the model accepts multimodal inputs at test time and has been trained using multimodal data to ensure compatibility with the test-time scenario. If a unimodal model were used at test time, it would lack the capability to leverage the full set of modalities present in the multimodal inputs, thereby limiting its performance and effectiveness in such scenarios.
To address the scenario you mentioned, where a unimodal model (e.g., a video model) is used for multimodal data, a finetuning stage would typically be required. For instance, the audio backbone could be initialized with the weights from video pretraining and then finetuned on audiovisual data. However, in our method, we do not perform any finetuning. Instead, we assume that the models have already been trained on multimodal data, which aligns with the scope and assumptions of our approach.

---

On mitigating performance drops at 100% missing modality

Thank you for your insightful comments. Adapting a model to a fully unimodal stream (100% missing rate) at test time is indeed a challenging scenario, particularly without labeled data. In this extreme case, MiDl neither degrades nor improves baseline performance, maintaining the integrity of the original multimodal model.
While methods like SHOT provide slight improvements over the baseline under a 100% missing ratio, they exhibit significantly lower performance compared to MiDl when some multimodal samples are available. MiDl is designed with the assumption that some presence of complete modalities at test time is necessary for effective adaptation, which aligns with the typical expectations for multimodal models.
We view it as a strength of MiDl that it avoids degrading the original model’s performance in this extreme case, rather than a limitation. Moreover, we highlight that MiDl demonstrates a significant performance boost during long-term adaptation, even if the test stream becomes unimodal over time (see Table 2 for detailed results).

---

On expandability to more modalities

We thank the reviewer for this insightful comment. MiDl does not impose any inherent limitations on the number of modalities; the scalability depends on the capabilities of the base model used. In Section 3, we formulated our approach m ∈ {A, V, AV} to align with our experiments on audiovisual egocentric datasets. However, the underlying problem and methodology can naturally extend to any number of modalities.
Similarly, MiDl is designed to work seamlessly with an arbitrary number of modalities. The formulations in Equation 1 and Equation 2 can be easily generalized by replacing AV with combinations of additional multimodal inputs, enabling broader applicability beyond the audiovisual setup presented in this work.

---

On dropping the secondary modality

We thank the reviewer for this comment. We report results for scenarios where the secondary modality is dropped in Section 6.2 and Table 4 of the main manuscript. Specifically, we present results for Epic Sounds when the video modality is dropped and for Epic Kitchens when the audio modality is dropped. These experiments demonstrate the robustness of our method across different modalities under varying missing probabilities.

---

On reproducibility of results

We are working on a code release, please stay tuned for future replies. We are committed to submit it before the discussion period ends.

Introduction

Motivation and Missing Modality Examples
We thank the reviewer for their thoughtful feedback and detailed suggestions. The necessity of addressing this problem can indeed be observed in practical scenarios such as the large-scale egocentric data collection in the Ego4D dataset (Grauman et al., 2022). In this dataset, while RGB video is consistently available, audio is absent in approximately 30% of the data due to privacy regulations in specific locations. Additionally, the reviewer’s suggestion about device failure or sensor deactivation is a valid and practical scenario. We also appreciate the reviewer’s point about the relevance of using cheaper modalities. This topic has been explored in recent literature (Grauman et al., 2023), where researchers focus on scenarios where models can infer information using fewer, less costly modalities to reduce energy consumption.

---

Contributions and Figures

Contributions and Figure 1 Improvements
We thank the reviewer for their suggestions regarding the figures and contributions. We have updated Figure 1 and the contributions statement accordingly in the manuscript. To clarify, the "potential performance trajectory" in the original figure was intended to represent the desired outcome of a successful adaptation strategy, serving as a conceptual illustration rather than being based on actual data. However, we agree that using our method's results instead makes the figure more impactful and better supports the message. We replace this plot with MiDl results on the Epic-Kitchens dataset.

---

Proposed Solution

Notation and KL Divergence Clarifications
We thank the reviewer for raising these points.

1. Regarding Notation: Indeed, $m$ is a discrete random variable sampled from $\{A, V, AV\}$, defined by probabilities $\{p_A, p_V, p_{AV}\}$, a property of the stream $\mathcal{S}$. For instance, $p_{AV} = 1$ implies $\mathcal{S}$ always reveals complete modalities. Sampling $x \sim \mathcal{S}$ equips $x$ with $m = M$. We are happy to revise the notation further if necessary.

2. Regarding KL Divergence: When $x$ is modal-complete, we define the KL divergence expectation over $m$ with $p_A = p_V = p_{AV} = \frac{1}{3}$.

3. Regarding Updates: As outlined in the interaction (gray box) in Section 4 (lines 262-263), MiDl adapts the model on the modal-complete data points while predicting without adaptation on other samples. Thus, MiDl produces predictions for every single data point regardless of its modality. As noted in lines 265-265, our work focuses on multimodal settings, and we assume $p_{AV} \neq 0$. We have modified the Takeaways subsection of Section 4 to more explicitly highlight this assumption.

4. Clarification on x_t: We apologize for any confusion regarding this notation. As clarified in lines 179–181, x_t refers to a sample or batch presented to the model at time step t. This does not imply all samples accumulated up to step t (i.e., x_0, x_1, \ldots, x_t​); rather, it strictly refers to the data arriving at time step t alone.

---

Experiments

Reproducibility Enhancements
We appreciate the reviewer’s request for additional details.

1. Hyperparameter Tuning: The implementation details for MiDl, including the selected hyperparameters, are provided in Section B.1. These hyperparameters were determined through a grid search to identify the optimal settings for the task. We will ensure that this clarification is made explicit in the manuscript. Additionally, our code release will further facilitate the reproducibility of these results.

2. Dataset Splits: We adhered to the official train/val/test splits provided for the Epic-Kitchens and Epic-Sounds datasets. The approximate ratios for these splits are 75% for training, 10% for validation, and 15% for testing. We will revise the manuscript to explicitly state these proportions to avoid any ambiguity.

Clarification on LTA Setup

We thank the reviewer for this thoughtful observation. To clarify, as outlined in Section 5.3, we reserve a subset of the training data for the Long-Term Adaptation (LTA) stage. The model observes labeled data from S_{in} prior to test time (during training), but we do not use any labels—whether from S_{in} or elsewhere—during the adaptation phase. This design simulates a practical scenario where a portion of training data can be stored and utilized for adaptation at test time without relying on labels. We do not use validation or test data for LTA because our assumption is that data arrives as a stream at test time, requiring immediate predictions. While our current setup reflects this realistic streaming assumption, in practical scenarios, one could envision access to test data in advance, allowing for storage of unlabeled data for long-term adaptation. This flexibility could further enhance the applicability of MiDl in various deployment settings.

---

Exploring Missing Modality Ratios
We thank the reviewer for these insightful observations. This scenario is indeed valuable to explore. As noted in Appendix B.4 and Table 9, we report results for the mixed missing modality setup, where either modality may be absent. These results demonstrate that MiDl consistently outperforms the baseline under all tested conditions, including scenarios with mixed modality availability.

Unimodal Clarifications
We appreciate the reviewer’s comments on clarifying the meaning of "Unimodal." In our manuscript, unimodal refers to a model that uses only the always-present modality (e.g., video for Epic-Sounds and audio for Epic-Kitchens). We apologize for the confusion caused by the presentation in Table 1, where the unimodal result appears only at the 50% missing rate. To clarify, the unimodal results are constant across all missing rates, as the model relies solely on the non-missing modality, which remains unaffected by the missing rate of the other modality. To avoid redundancy, we initially reported the unimodal result once in the middle of the table, but we acknowledge that this presentation may have caused confusion. We have revised the manuscript to explicitly show the unimodal results across all missing rates for clarity.

---

Architecture Diversity

We thank the reviewer for this insightful comment. We would like to clarify that we do present results with different architectures and models. Specifically, we report results for self-attention-based models in Section 6.1 and Omnivore in Section 6.2. While we agree that further exploration with more diverse setups (e.g., convolutional backbones) could be valuable, our focus was on evaluating state-of-the-art and widely-used architectures, which are predominantly transformer-based. We appreciate the suggestion to reorganize Table 3 to present results with different architectures under a fixed missing rate. While our current presentation emphasizes performance across varying missing rates, we recognize that including architecture-level comparisons could provide complementary insights. We are committed to releasing the code, which we hope will enable further exploration of this problem from an architectural perspective.

---

Computational Efficiency

TTA vs. Retraining
We apologize for any confusion. As mentioned in Section 4 (Lines 270–275), we formulate the missing modality challenge within the test-time adaptation scenario. In this framework, we make no assumptions about the training process. Instead of retraining the network, we adapt it at test time by updating only a small subset of parameters.

Omnivore experiment Our approach is designed to work with existing pretrained models, as demonstrated in our experiments, including the Omnivore pretraining test. This emphasizes the practicality of MiDl, as it eliminates the need for retraining on large datasets, aligning with our primary motivation.

---

Minor Points

Bolded Numbers Table 6
We thank the reviewer for catching this oversight. The correct value (55.2 in the "Dl" column) has been bolded in the updated manuscript.

The authors have done a fairly good job in their rebuttal, addressing my suggestions and making reasonable changes to the work. However, I remain unconvinced regarding the privacy-related motivation and the suggested computational benefits.

While Ego4D redacted modalities due to privacy concerns, this was done for training purposes and model construction. It's unclear why a modality would be missing for privacy reasons when deploying the model in a real-world setting. This may be a misunderstanding or misphrasing in the introduction.

Regarding computational efficiency, I think the authors misunderstood my original stated weakness. The work mentions that "MiDl is 5x more expensive" during testing. This raises the question: if the test set is more than one-fifth the size of the training set, wouldn't retraining a different model be faster than adapting with MiDl? I believe more analysis is needed to convincingly demonstrate that MiDl TTA is computationally superior to retraining.

Nevertheless, this is the first work I've encountered that explicitly explores missing modality TTA. Although there's room for expansion in terms of the number of modalities used, datasets employed, and baselines compared against, I believe the work provides a modestly sufficient contribution for ICLR. The exploration of various related and interesting aspects of this problem, such as different missing rates and LTA, is also noteworthy.

Consequently, I am revising my score to marginally above acceptance.

On comparison with related works addressing the missing modality problem

We thank the reviewer for pointing out these related works. We would like to clarify that the proposed methods primarily address the missing modality problem during training. For example, Dai et al. investigate a strategy of randomly dropping video frames during training to improve the robustness of a multimodal system. Similarly, Lee et al. propose a method to train a network capable of generating audio features to handle missing modalities. Wang et al. focus on a multimodal learning approach that models shared and specific features for classification and segmentation tasks. In contrast, our work formulates the missing modality problem as a test-time adaptation challenge, a novel perspective that assumes no access to the training process or labels and instead addresses the problem entirely at test time. This distinction fundamentally differentiates our approach from the works cited, as our focus is on adapting trained models dynamically to optimize performance in the face of missing modalities. We have added these references in the main manuscript.

As part of this framing, we compare MiDl against existing test-time adaptation methods, which are more aligned with the assumptions and constraints of our setup. Nonetheless, we appreciate the reviewer’s suggestion and will ensure these works are acknowledged in the related work section, highlighting the distinctions between training-time and test-time approaches to the missing modality problem.

---

On enriching the benchmark with additional TTA methods

We thank the reviewer for their valuable suggestions regarding enriching the benchmark with additional TTA methods. Our work already compares MiDl to several commonly used TTA methods to validate its effectiveness, including Tent (Wang et al.) and ETA (Niu et al.), which are explicitly mentioned in the manuscript. Results for these methods are presented in Tables 1, 3, and 7, showcasing their performance under different scenarios and comparing them to MiDl.

The primary goal of our work is to redefine the missing modality problem as a test-time adaptation challenge, introducing a novel approach where pretrained models are adapted at test time to optimize performance in the face of missing modalities. We conduct extensive experiments, including ablations across various scenarios such as different backbones, pretraining strategies, and modality setups, to demonstrate MiDl’s effectiveness.

While we appreciate the suggestion to include additional methods like Contrastive TTA (Chen et al.) and Robust TTA in dynamic scenarios (Yuan et al.), we emphasize that our current comparisons and analyses already provide a comprehensive evaluation of MiDl’s performance. Future work could further expand on these comparisons to include additional methods for broader validation.

---

On providing qualitative results and failure case analysis

Thank you very much for this valuable suggestion. We are currently preparing qualitative examples comparing our approach with the baseline. These examples, along with an analysis of failure cases, will be included in the supplementary material in the revised submission.

---

On generalizability beyond Epic-Kitchens

Thank you for pointing this out. As mentioned in Section 5.1, we validate our approach on two distinct datasets: Epic-Sounds (Huh, et. al., 2023) and Epic-Kitchens (Damen, et. al, 2020). To align with the experimental setup of prior work, we assume different missing modalities for each dataset, with video missing in Epic-Kitchens and audio missing in Epic-Sounds. This demonstrates the adaptability of our method to varying modality configurations.

---

On TSNE visualization of latent space

We thank the reviewer for this insightful suggestion and their interest in understanding the effects of our method. While TSNE visualization is commonly used to illustrate feature learning and the clustering behavior of learned representations, we would like to emphasize that MiDl is not a feature learning approach in the traditional sense. Instead, it focuses exclusively on adapting pretrained models at test time by updating only the parameters of the normalization layers to handle missing modalities dynamically.

This design choice means that MiDl does not aim to significantly alter the learned feature space but rather adjusts the model's predictions to maintain robustness under test-time conditions. Consequently, the use of TSNE to visualize changes across epochs may not be directly relevant to evaluating MiDl’s effectiveness.

If the reviewer has specific aspects of the latent space or adaptation process they would like to see explored, we would be happy to incorporate such analyses to further enhance the interpretability of our method.

On the assumption and phrasing regarding modality invariance

We thank the reviewer for raising this insightful point. We would like to clarify the distinction between our setup and the modality distillation problem, as they address fundamentally different challenges. In modality distillation, the primary goal is to transfer knowledge from a teacher model, which is usually trained with access to all modalities, to a smaller, typically unimodal student model. The focus is on training a student model to approximate the teacher’s performance despite having access to fewer (or weaker) modalities.

In contrast, our approach assumes that the model should inherently possess the capability to make consistent and accurate predictions, irrespective of the available modality or combination of modalities. For example, whether the model observes a silent video of a bird, hears the chirping sound alone, or has access to both, it should consistently recognize that it is observing a bird. This is not a process of distillation from one model to another but rather an effort to provide a single, modality-agnostic model that learns to generalize across different modality combinations.

---

On computational overhead and latency in real-world applications

We thank the reviewer for highlighting this important consideration. We would like to emphasize that MiDl is agnostic to the base model, meaning it makes no assumptions about the model architecture or pre-training strategy. Consequently, our method can be applied even if the pre-trained model has already been compressed or distilled.

We acknowledge the computational cost associated with test-time updates, as noted in Section 6.5. This trade-off is a common consideration for any test-time adaptation methods and should be weighed against the performance benefits in real-world deployments.

---

Extra experiments in online scenarios

Thank you for this valuable suggestion! We agree that addressing more complex online scenarios would be a compelling direction for further exploration. In fact, we followed the recent literature on online test-time adaptation (Alfarra et al., 2023) to define our stream setting, ensuring alignment with the current state-of-the-art. We are open to exploring additional scenarios that could showcase the applicability of our approach in even more complex online settings. Could you please elaborate on the specific scenarios or challenges you believe would better demonstrate the utility of our method? We would greatly appreciate your input.

---

On extreme examples with misaligned modalities

We appreciate the reviewer’s curiosity and for bringing up this interesting scenario. However, we would like to clarify that this situation is beyond the scope of our paper. The scenario you describe pertains to misalignment in multimodal data, which differs fundamentally from the missing modality problem that we address. In the missing modality problem, we are aware that a modality is absent, which may occur due to device malfunctions, sensor deactivation for privacy or efficiency, or similar reasons, as discussed in the introduction.

In the misalignment scenario you outlined, all modalities are still present but are not semantically aligned. If the misalignment is intentional, such as injecting incorrect inputs, it may fall under the category of adversarial attacks. This is outside the focus of our work, as we concentrate on scenarios where a modality is simply missing and not replaced with deliberately crafted or noisy inputs.

Even in less extreme cases of natural misalignment—such as a TV playing unrelated sounds in the background while the video reflects what a person sees—this situation involves all modalities being available and presents a different setup to the one tackled by our work. Our focus remains on the challenges and solutions specific to missing modality scenarios.

---

On adding blocks or partial updates

Thank you for this interesting suggestion. Efficiency and latency are indeed critical considerations when applying test-time adaptation methods. As mentioned in Section B.1 (Lines 735–739 of previous document, or 751-755 on updated document) and Section B.3, our approach follows the prior line of work in test-time adaptation methods by only updating the normalization layers when applying MiDl, which reduces computational overhead.

Additionally, it is important to note that as we mentioned in Section 6.1, our approach is architecture-agnostic. This flexibility allows users to opt for a more lightweight architecture equipped with MiDl to tailor their specific application, thereby further addressing concerns around efficiency and latency.