We thank the reviewer for their thoughtful review. We are glad that the paper’s novelty and the use of VMamba’s intrinsic traversal mechanism were well appreciated. The recognition of our permutation strategy as a form of implicit augmentation, along with the strong empirical results, reinforces the effectiveness and originality of the proposed approach.

1. The experiment design only focuses on the simplest TTA setting. Consider Continual TTA [7] or temporal correlation settings [6, 8, 9].

We would like to clarify that continual test-time adaptation and temporal correlation represents distinct subfields within test-time adaptation, with different assumptions and objectives, often requiring longer adaptation timelines and specialized setups. Additionally, most papers in the TTA literature do not compare their results against continual adaptation or temporally correlated test-time settings, such as those explored in [1, 2, 3]. Moreover, implementing and evaluating TRUST in this setting would require extensive design and tuning beyond the scope of a one-week rebuttal period. We will explore this in future works.

2. What's the latency when each batch requires all three phases (offline, adaptation, evaluation) to be performed?

Our method consists of three stages: Offline phase: We evaluate our pre-trained model under different traversal permutations and select the top-k permutations with the lowest entropy values, corresponding to the most confident predictions. Since no model adaptation occurs at this stage, the procedure is equivalent to a forward-pass evaluation and is computationally lightweight. For instance, on the PACS dataset, evaluating for a single permutation takes only 4 seconds for all images in this dataset. Extending this across all 24 permutations results in a total evaluation time of 1.5 minutes, which is a one-time cost. Our experiments show that the top-K permutations remain consistent across all test datasets, as visualized in Figure 2 of the supplementary material. This consistency stems from the similarity of these permutations to the original traversal order used during pre-training, as reflected in their high confidence scores. Therefore, this procedure can be performed only once on a single dataset to determine the appropriate Ks, and then used across all test datasets. Adaptation stage: We measured the latency for the batch size of 128 used in our experiments for forward and backward passes, which results in approximately 1.2 second per permutation. Evaluation stage: In this phase, we first perform weight averaging, which involves a simple averaging of the model weights across selected permutations (0.018 second). The subsequent evaluation functions similarly to the offline stage, no adaptation is performed, and we simply test the model (0.2 second). Our overall per permutation is: 1.2 + 0.018 + 0.2 = 1.41 second.

3. Could the proposed method be applied to other tasks like segmentation?

We conducted additional experiments on segmentation tasks using various datasets, including Pascal VOC21 and Pascal Context 59. The results demonstrate that TRUST performs well in segmentation, outperforming methods like TENT. This further supports the generalizability and effectiveness of our approach beyond classification settings.

V21:

P59:

4. Is it really necessary to develop a method specialized for it?

As we discussed in the introduction, Mamba is emerging as a powerful alternative to both CNNs and ViTs across a range of vision tasks. Unlike ViTs, which rely on self-attention and quadratic complexity with respect to sequence length, Mamba is built on SSMs that enable linear-time sequence modeling, making it highly efficient for long input sequences. This makes Mamba especially attractive in settings with limited compute resources. Despite being more lightweight, Mamba retains competitive or even superior performance in many tasks. Several recent papers [4, 5, 6] have shown that Mamba outperforms CNNs and ViTs on image classification, segmentation, and even video understanding, while using fewer parameters and less memory. Previous test-time adaptation methods, such as TENT and SHOT, are not well-suited for Mamba-based models, as demonstrated in Table 2 of our paper. Due to the unique architecture of Mamba and favorable properties, such as linear-time complexity, we focused on designing a strategy specifically tailored for Mamba to effectively handle domain shifts during test time in vision applications.

5. The adaptation exclusively targets "Mamba-specific state space parameters." How does the performance differ when other components (e.g., normalization layers, as in TENT) are adapted?

We provided this experiment in table 2 of our supplementary material. The results represent that our method outperforms TENT in both settings, by 3.1% when using BatchNorm adaptation and by 11.8% with SS2D adaptation. We opt to proceed with SS2D adaptation, as traversal permutations directly influence Mamba-specific parameters encoded in the SS2D blocks. Updating these parameters is therefore essential to fully capture the effects of traversal-based modifications.

ImageNet-C:

6. Compare with the suggested baselines.

Due to architectural differences between VMamba and other backbones used in the baselines mentioned by the reviewer, a direct comparison would not be entirely fair. These methods, which rely on ViT-based architectures, will be included in a separate category in the final version of the paper. However, to provide a fair evaluation, we adapted the ViT-based method from [2] by replacing its backbone with VMamba and modifying it to be compatible with VMamba backbone. Specifically, we replaced the CLS token (absent in VMamba) with the mean of all tokens and adjusted the number of learnable tokens to match VMamba’s dimensionality. This ensures that both methods operate under the same architectural constraints. Our experimental results demonstrate that TRUST, which is specifically designed to exploit VMamba’s traversal mechanism, significantly outperforms the adapted baseline, highlighting its robustness under distribution shift.

ImageNet-C:

7. Are the traversal permutations fundamentally different from rotating the original image?

While image rotation changes the spatial arrangement of tokens, it is not equivalent to the traversal permutations we apply within VMamba, as the specific traversal orders used in our method cannot be replicated through simple rotations. As shown in Figure 4 of our paper, replacing traversal permutations with standard image rotations leads to significantly lower performance. This confirms that the gains from TRUST arise specifically from manipulating the causal processing order through traversal permutations, rather than from altering the input appearance via rotation.

[1] WATT: Weight average test time adaptation of CLIP. Advances in neural information processing systems, 37, 48015-48044.

[2] Clipartt: Adaptation of clip to new domains at test time. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 7092-7101). IEEE.

[3] Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35, 14274-14289.

[4] Vmamba: Visual state space model. Advances in neural information processing systems, 37, 103031-103063.

[5] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model.

[6] LocalMamba: Visual State Space Model with Windowed Selective Scan. CoRR.

We thank the reviewer for their thoughtful feedback and are pleased that TRUST was recognized as a novel and effective TTA method for Mamba-based vision models. We appreciate the positive remarks on our use of traversal dynamics, strong experimental results across diverse benchmarks, and the clarity of our ablation studies. Below, we address the reviewer comments in detail.

1-1. Time complexity of the traversal permutation generation and selection in the offline phase.

In the offline phase, we evaluate our pre-trained model under different traversal permutations and select the top-k permutations with the lowest entropy values, corresponding to the most confident predictions. Since no model adaptation occurs at this stage, the procedure is equivalent to a forward-pass evaluation and is computationally lightweight. For instance, on the PACS dataset, evaluating for a single permutation takes only 4 seconds for all images in this dataset. Extending this across all 24 permutations results in a total evaluation time of 1.5 minutes, which is a one-time cost. Our experiments show that the top-K permutations remain consistent across all test datasets, as visualized in Figure 2 of the supplementary material. This consistency stems from the similarity of these permutations to the original traversal order used during pre-training, as reflected in their high confidence scores. Therefore, this procedure can be performed only once on a single dataset to determine the appropriate Ks, and then used across all test datasets.

1-2. How does the scalability of this component affect the practical application of TRUST, especially with larger models or increased number of permutations?

As mentioned previously, each permutation in the offline phase is evaluated using only a forward pass, no backpropagation is involved. The total evaluation time scales linearly with the number of permutations. In terms of memory, the model size remains constant regardless of the number of permutations, since the same model is reused for each forward pass. Therefore, the memory required is simply that needed to run a single evaluation of the model.

2. Elaborate on the potential challenges or modifications required to adapt to other NNs such as CNNs or ViTs.

Mamba is emerging as a powerful alternative to both CNNs and ViTs across a range of vision tasks. Unlike ViTs, which rely on self-attention and quadratic complexity with respect to sequence length, Mamba is built on SSMs that enable linear-time sequence modeling, making it highly efficient for long input sequences. This makes Mamba especially attractive in settings with limited compute resources. Despite being more lightweight, Mamba retains competitive or even superior performance in many tasks. Several recent papers [1, 2, 3] have shown that Mamba outperforms ViTs on image classification, segmentation, and detection, while using less memory.

Previous test-time adaptation methods, such as TENT and SHOT, are not well-suited for Mamba-based models, as demonstrated in Table 2 of our paper. Due to the unique architecture of Mamba and favorable properties, such as linear-time complexity, we focused on designing a strategy specifically tailored for Mamba to effectively handle domain shifts during test time in vision applications. Extending our approach to CNNs or ViTs would undermine our core objective, as it would forgo the key advantage of leveraging Mamba’s efficient and sequential computation.

3-1. Deeper theoretical explanation of why these confidence paths consistently lead to better generalization, beyond the current explanation of exploring flatter minima?

We provide a deeper theoretical justification based on SWAD [5]. In our setting, each traversal permutation $\pi_i$ defines a distinct causal ordering through which VMamba processes input patches, resulting in different trajectories of hidden states $\{\mathbf{h}(t)\}$. Low-entropy (high-confidence) permutations consistently correspond to stable recurrent dynamics, where the influence of corruptions is minimized. This stabilizes the hidden-state evolution, yielding low-variance predictions and reducing sensitivity to domain-specific perturbations. From a loss landscape perspective, such paths correspond to regions of low curvature, as smoother predictions typically reflect flatter neighborhoods around $\theta_i$ (parameters of adapted model with $\pi_i$).

Therefore, selecting low-entropy permutations acts as a proxy for identifying solutions that are both dynamically stable (in terms of hidden-state recurrence) and geometrically robust (in terms of curvature). Averaging over the adapted weights from these confidence paths further concentrates the model in a flat, low-loss region of parameter space. This synergy between entropy-based selection and weight averaging aligns directly with the generalization theory of SWAD [5], and provides a principled explanation for the observed robustness of TRUST across domains.

3-2. Sensitivity of the results to the choice of the hyperparameter K (number of top permutations selected) and the methodology used to determine its optimal value?

By applying multiple traversal permutations, we expose the model to diverse causal perspectives of the same input. This variation encourages the model to learn different adaptation patterns at test time. Increasing the number of traversal permutations introduces more variation and, as shown in works such as Model-Soups [4], a greater number of diverse, correctly-aligned representations (our top-K permutations) can enhance the effectiveness of weight averaging. Figure 5 in the paper (table below) empirically supports this observation across three datasets. Our experiments shows that increasing permutation diversity improves robustness, with performance gains saturating around six permutations. This can offer a good trade-off between accuracy and efficiency.

Limitations

Thank you for the thoughtful suggestions. We will incorporate a dedicated Limitations section in the final version, addressing potential societal impacts, including bias, adversarial robustness, ethical considerations, and environmental footprint.

[1] Vmamba: Visual state space model. Advances in neural information processing systems, 37, 103031-103063.

[2] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model.

[3] LocalMamba: Visual State Space Model with Windowed Selective Scan. CoRR.

[4] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning (pp. 23965-23998). PMLR.

[5] Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34, 22405-22418.

Dear Reviewer sdHM,

Thank you very much for your thorough and insightful review. We hope that our rebuttal has addressed your concerns, especially regarding the computational cost and scalability of permutation selection, the theoretical basis for entropy-guided traversal selection, and the sensitivity to hyperparameters such as the number of top-K permutations. We would be happy to provide further clarification or discussion on any remaining points you might have.

We thank the reviewer for their thoughtful review. We are glad our paper was recognized as novel and practical in addressing VMamba's limitations under distribution shifts. We appreciate the positive comments on the method's effectiveness, its source-free nature, and the clarity of the writing.

1. Explanation of how the simple weight averaging across multiple models leads to performance gains.

The concept of weight averaging seeks to identify a solution in parameter space that lies within a flat region of the loss landscape. Such flat minima are associated with low loss values that remain stable under small perturbations to the model parameters. Models converging to these regions tend to generalize better under distribution shifts, as their performance is less sensitive to noise or minor changes in input data and parameters. In our approach, the integration of traversal permutations with weight averaging is inspired by Robust Risk Minimization (RRM) [1]. RRM aims to minimize the worst-case empirical loss within a neighborhood around the current parameters, formulated as:

$E_{D}^{\gamma}(\theta) = \max_{\|\Delta\| \leq \gamma} \hat{E}_{D}(\theta + \Delta)$

where $\gamma > 0$ defines the neighborhood around the model parameters $\theta$, and $\Delta$ represents small perturbations. While we do not directly optimize this objective, the principle aligns with our use of weight averaging across traversal permutations. By averaging the adapted weights, we converge toward a flatter region in weight space, improving generalization during test-time adaptation and enhancing robustness to distribution shifts.

2. How is the value of K (i.e., the number of traversal permutations or model variants) determined across different images or datasets? Is K fixed, adaptive, or tuned per dataset or image?

As shown in Figure 5 of the paper (table below), we experimented with different values of K across multiple datasets. Prior work such as Model Soups [4] suggests that averaging a diverse set of well-aligned models (our top-K permutations) can improve performance. Consistent with this, our results indicate that accuracy generally increases with more permutations, but the gains saturate beyond K = 6. This suggests that using six permutations provides an effective balance between accuracy and computational cost. Accordingly, we fixed K = 6 for all experiments.

3. Is your method applicable to other vision tasks such as object detection or semantic segmentation?

Thank you for this thoughtful question. We conducted additional experiments on segmentation tasks using various datasets, including Pascal VOC21 and Pascal Context 59. The results demonstrate that TRUST performs well in segmentation, outperforming methods like TENT. This further supports the generalizability and effectiveness of our approach beyond classification settings.

V21:

P59:

[1] Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34, 22405-22418.

Dear Reviewer pWep,

Thank you very much for your thoughtful review and constructive feedback. We hope our rebuttal has addressed your questions, particularly regarding how weight averaging contributes to improved generalization, how the top-K value is selected consistently across datasets, and the applicability of TRUST to other vision tasks like segmentation. We would be glad to address any further questions or feedback you may have.

We thank the reviewers for the valuable comments and are pleased that TRUST was recognized as a novel and well-motivated method for Mamba-style architectures. We appreciate the positive remarks on the clarity of our writing, the effectiveness of our approach, and the depth of our experiments and ablations. Below, we address the reviewer comments in detail.

1-1. Why is the model performance sensitive to traversal orders?

State Space Models (SSMs) are inherently direction-sensitive, meaning that the order in which input patches are traversed has a significant impact on model performance. According to the Mamba formulation (Eq. 1 of our paper), the hidden state at each step ($h(t-1)$) directly influences the next ($h(t)$). Therefore, the choice of traversal order propagates its effects through the entire sequence of hidden-state updates. This directional dependency also comes from the fact that the hidden states learned during training are aligned with the original traversal order. When a different patch order is used at test time, the model is exposed to a sequence it has not encountered before, resulting in a mismatch between learned and test-time dynamics. As noted in the Mamba paper, this directional bias can cause the model to encode information more effectively along the original traversal path.

$h(t) = A h(t-1) + B x(t), \quad y(t) = C h(t)$

1-2. How the top-k traversal orders are selected without labels?

We emphasized that our selection of the top-k traversal permutations is entirely unsupervised and does not require access to labels. As detailed in Section 3.2 and illustrated in Figure 1 (offline phase) of the paper, we rank traversal orders solely based on the average Shannon entropy of predictions on a test dataset. We conducted experiments on ImageNet-C and selected the top-k traversal permutations based on the lowest prediction entropy. We observed that these top-k permutations remained largely consistent across different test data distributions. This finding is illustrated in Figure 2 of the supplementary material.

1-3. Find a reliable heuristic to choose the best test-time permutation

To identify the most effective test-time permutation, we adopt Shannon entropy as a self-supervised confidence metric. While low entropy does not strictly guarantee better accuracy, it serves as a strong proxy for model certainty. In particular, entropy captures the sharpness of the predicted probability distribution, lower entropy indicates more confident predictions, which often correlates with more reliable traversal orders. Therefore, entropy offers a useful heuristic that helps rank permutations by their potential to produce accurate and stable outputs in the absence of labeled test data.

2-1. Computational Complexity

Our method supports an efficient parallel adaptation strategy, which we represented in Figure 1 of supplementary material and we discussed its computational overhead in the ablation study of the paper. In this mode, we handle K traversal permutations simultaneously. One batch is first split into K subsets, each corresponding to a different permutation. Each subset is then passed to an independent SS2D TRUST Version block, where the SS2D parameters are adapted in parallel while the rest of the model remains shared across all paths. After adaptation, the outputs are concatenated back into a single batch, which allows for efficient GPU utilization. For evaluation, we perform a weight averaging step across all adapted SS2D modules, producing a single unified SS2D TRUST Version block. This averaged block is then used for inference on the full batch.

2-2. Could the combination of augmentation and TENT yield comparable performance?

We explored the use of different augmentations as an alternative to multiple traversal permutations in Figure 4 of the paper. That experiment evaluates the impact of weight averaging across models adapted with various augmentations versus those adapted with different traversal permutations. To ensure a fair comparison, we match the number of data augmentations to the number of traversal permutations. As shown in the table below, different traversal permutations (TRUST) consistently outperforms different augmentations, highlighting the superior effectiveness of permutation diversity in enhancing the model performance.

CIFAR10-C:

Dear Reviewer GHW6,

Thank you sincerely for your thoughtful and detailed review. We hope that our rebuttal has addressed your concerns, particularly those regarding (1) the sensitivity of model performance to traversal order and the rationale behind entropy-based selection, and (2) the computational trade-offs, including our proposed parallel adaptation strategy and comparative analysis with augmentation-based TTA methods. We would be happy to further elaborate on any remaining questions or suggestions you might have.