RobIA: Robust Instance-aware Continual Test-time Adaptation for Deep Stereo

We appreciate the reviewer VsX9 for recognizing the significance of our CTTA framework for stereo depth estimation, the motivation behind our MoE and AdaptBN-based components, and the clarity and thoroughness of the manuscript and experiments.

---

Weakness A. Applicability to Other Architectures

Yes, our method is applicable to frameworks such as HITNet [A] and TemporalStereo [B], as both AttEx-MoE and AdaptBN-Teacher are designed as plug-and-play modules compatible with a wide range of CNN-based stereo architectures.

AttEx-MoE applies instance-aware, channel-wise excitation exclusively within the feature extractor, as this stage is critical for capturing high-quality, geometry-aware features for stereo matching. Its row-wise self-attention router operates along epipolar lines and naturally aligns with the geometry of rectified stereo images, making it broadly applicable to architectures that process stereo pairs. We elaborate further on this benefit in the Weakness D. Why Epipolar Line? section.

Similarly, AdaptBN is applicable to any CNN networks using batch normalization. We provide further discussion in the Weakness C. Substitutes for Batch Normalization section.

---

Weakness B. Applicability to Transformer-based Architectures

Yes, AttEx-MoE is compatible with Transformer-based stereo architectures.

While our experiments focus on CNN backbones (e.g., CoEx), AttEx-MoE is backbone-agnostic. Its components, the row-wise self-attention router and Sigmoid gating, operate on feature maps, and can be applied to any encoder as long as it maintains image's spatial feature structure.

In Transformer-based stereo models [C] [D], token sequences can be reshaped into feature maps, enabling seamless integration of AttEx-MoE into the encoder without architectural changes.

---

Weakness C. Substitutes for Batch Normalization

AdaptBN is not inherently tied to BatchNorm but is based on adapting a small set of affine parameters within a frozen backbone. While we do not experiment with other normalization schemes in this work, the underlying principle is conceptually generalizable:

• GroupNorm / InstanceNorm: Like BN, both provide per-channel affine parameters $(\gamma, \beta)$, suggesting that similar adaptation (e.g., AdaptGN or AdaptIN) could be feasible.

• LayerNorm (e.g., in ViT): Despite operating over hidden dimensions, its affine parameters $(\gamma_{LN}, \beta_{LN})$ may also be adapted per layer to enable token-level adaptation.

For backbones without explicit affine layers (e.g., Conv + ReLU), a lightweight 1×1 'scale-shift' adapter (two parameters per channel) can be inserted with minimal overhead.

In summary, AdaptBN’s effectiveness stems from adapting affine modulation parameters, not BatchNorm specifically. This approach may be extended to other architectures with compatible parameterizations. We will clarify this in the revised manuscript.

---

Weakness D. Why Epipolar Line?

While full self-attention is effective for modeling global dependencies, it incurs prohibitive computational cost. In rectified stereo geometry, corresponding points across view lie along the same image row, making horizontal context more relevant than vertical context for disparity estimation [C]. We thereby adopt row-wise self-attention, which focuses along epipolar-consistent scanlines. This preserves disparity-relevant context while reducing the attention complexity from $H \times W \times H \times W$ to $H \times W \times W$, lowering memory and FLOPs by a factor of $H$. Thus, row-wise attention is not merely a speed heuristic, but a geometry-aware inductive bias that enhances both efficiency and performance. Empirical results in Tab. 4 support this design: among various router alternatives, row-wise attention achieves the best performance.

---

About Paper Formatting Concerns

We appreciate your careful reading and pointing out the formatting issues. We will fix the boldface formatting in Table 1 and the other tables.

---

Citations

• [A] Tankovich, Vladimir, et al. "Hitnet: Hierarchical iterative tile refinement network for real-time stereo matching." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
• [B] Zhang, Youmin, Matteo Poggi, and Stefano Mattoccia. "Temporalstereo: Efficient spatial-temporal stereo matching network." 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023.
• [C] Li, Zhaoshuo, et al. "Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
• [D] Weinzaepfel, Philippe, et al. "Croco v2: Improved cross-view completion pre-training for stereo matching and optical flow." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

We appreciate the reviewer 4p4z for recognizing the simplicity and efficiency of our CTTA framework, the realism of our benchmark design, and the clarity of our overall presentation.

---

Weakness 1. Comparison with MAD++

We thank the reviewer for highlighting the importance of a fair comparison with MAD++. To address this, we re-implemented our method, including AttEx-MoE and the AdaptBN teacher, on top of the MADNet2 [A] encoder, which is also used by MAD++ in Table 1, 2, and 8. Since MADNet2 lacks normalization layers, we inserted BatchNorm layers after each convolution to enable AdaptBN and maintain architectural consistency.

Table A presents CTTA results using the MADNet2 backbone along with respective runtime, as requested.

• On KITTI, which presents relatively mild domain shifts, (b) FULL++ achieves D1-all 1.13% by updating all layers, while (d) our AttEx-MoE method achieves a better result (1.12%) with fewer parameters. (e) RobIA, which combines AttEx-MoE with the AdaptBN teacher, maintains a strong KITTI score of 1.1%.

• On DrivingStereo, which exhibits stronger domain shifts, (b) FULL++ struggles with large domain shifts (6.13%), and (c) MAD++ provides only minor improvement (5.86%). In contrast, (d) our AttEx-MoE method further reduces the error to 5.53%, and (e) RobIA achieves the best result at 4.83%.

These results show that our input-aware AttEx-MoE, combined with dual-source supervision, matches full tuning on stable domains (KITTI), and significantly outperforms both full and modular adaptation approaches under large distribution shifts (DrivingStereo).

Table A

---

Weakness 2. Limited overall performance gain

RobIA’s Gains Are Significant Under Realistic Domain Shifts.

We appreciate the concern regarding the seemingly marginal improvements of RobIA on KITTI. We agree that the performance gap may appear small. However, we would like to emphasize that this is primarily due to mild domain shifts and dense SGM pseudo-labels on KITTI, which make it easier to adapt even with lightweight methods like AdaptBN. In contrast, under more realistic and challenging conditions such as weather variation (DrivingStereo) and nighttime scenes (DSEC), RobIA demonstrates substantial improvements over AdaptBN.

• In Tab. 1 and 2, (e) AdaptBN (0.03M trainable parameters) struggles under substantial domain shifts such as weather changes (DrivingStereo) and nighttime scenes (DSEC), yielding D1-all scores of 3.08% and 4.54%, respectively.

• In contrast, (h) AttEx-MoE w/ AdaptBN-Teacher (1.22M) significantly improves performance to 2.77% and 4.46%.

KITTI Performance Reflects Domain Simplicity, Not Method Weakness.

AdaptBN performs competitively on KITTI (0.91%, Tab.8), likely due to its (1) smaller domain gap (e.g., city $\rightarrow$ residential) and (2) higher SGM pseudo-label density (92%) compared to DrivingStereo (72%) and DSEC (45%). These factors suggest that KITTI presents a more favorable adaptation setting, where even lightweight methods like AdaptBN can perform well.

Sequential Adaptation, KITTI $\rightarrow$ DrivingStereo, Further Highlights RobIA's Advantage.

To further support this point, we simulated a sequential CTTA scenario where the model first adapts to KITTI for 5 rounds, followed by adaptation to DrivingStereo for another 5 rounds, mimicking a realistic progression from a stable to a more challenging domain. Results are shown in Table B.

• (b) and (c) show similar error rates with (a) on KITTI, but they achieve substantially lower error on DrivingStereo.

• (a), despite strong performance of 1.11% on KITTI, fails to adapt in the second phase (5.90%).

These results highlight the robustness of our AttEx-MoE design, which generalizes better under sequential, cross-domain conditions, mirroring realistic deployment settings.

Table B

---

Weakness 3. About Paper Formatting Concerns

We appreciate your careful reading and pointing out the formatting issues. We will correct the loss term and fix the boldface formatting in Table 1 and the other tables. Additionally, we will rename the 'w/ AdaptBN' label to avoid ambiguity.

---

Citations

• [A] Tonioni, Alessio, et al. "Real-time self-adaptive deep stereo." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

We thank the reviewer CJJ4 for their thoughtful feedback. We are encouraged that they found our idea innovative and novel, supported by extensive experiments and efficient use of limited resources.

---

Weakness 1. Broader Backbone Evaluation

We appreciate the suggestion regarding evaluation using additional competitive architectures. In response, we extended our experiments to two widely recognized and publicly available backbones:

• IGEV-Stereo [A]: a widely adopted iterative refinement model
• LightStereo [B]: a lightweight model with 2D cost aggregation

1-1. Evaluation on IGEV-Stereo

As shown in Table A, our method significantly improves performance. These results confirm that our plug-and-play modules effectively enhance generalization even on top-performing backbones.

• (a) IGEV-Source suffers from domain shift (7.26% D1-all on DrivingStereo).
• (b) AdaptBN tuning only BN parameters and (c) FULL++ still struggle for adaptation (3.87%, 3.56%, respectively).
• (d) Our AttEx-MoE reduces DrivingStereo error to 3.15%.
• (e) The proposed method, RobIA (MoE w/ AdaptBN-Teacher), achieves the best performance: 2.76% on DrivingStereo.

Table A

1-2. Evaluation on LightStereo

RobIA also generalizes well to LightStereo, a compact real-time model. The results are shown in Table B.

• (a) LightStereo-Source shows severe degradation under domain shift (18.59%).
• (b) AdaptBN improves accuracy (5.95%) but remains limited on dynamic scenes.
• (d) Our AdaptBN-Teacher improves CoEx FULL++ to 5.18%, reinforcing the strength of our dense pseudo-labeling strategy.
• (e) AttEx-MoE alone reduces DrivingStereo error to 5.76%.
• (f) The proposed method, RobIA, achieves the best results: 4.95% on DrivingStereo.

Table B

---

Weakness 2. About Paper Formatting Concerns

We appreciate your careful reading and pointing out the formatting issues. We will correct the loss term and fix the boldface formatting in Table 1 and the other tables.

Citations

• [A] Xu, Gangwei, et al. "Iterative geometry encoding volume for stereo matching." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.
• [B] Guo, Xianda, et al. "LightStereo: Channel Boost Is All You Need for Efficient 2D Cost Aggregation." arXiv preprint arXiv:2406.19833 (2024).

We appreciate the reviewer i8yG for highlighting the relevance of our test-time training setting, the novelty of our MoE-based design for CNNs, and our solution to performance degradation in invalid regions, along with the supporting experimental evidence.

---

Weakness a. Computational Cost

We respectfully point out that comparing (j) MoE w/ AdaptBN-Teacher to (d) CoEx Source in Tab. 3 is may not be entirely fair, as (d) represents a pure inference baseline without any adaptation, which falls outside the scope of test-time adaptation. While (j) incurs higher latency, it achieves the best overall performance (D1-all 2.77%). More importantly, (i) our MoE-only variant strikes a favorable balance between performance and efficiency, achieving strong accuracy (D1-all 2.98%) with moderate runtime (97.34ms). Practitioners may choose between (i) and (j) depending on the desired trade-off.

---

Weakness b. How does the MoE architecture help generalization?

We appreciate the reviewer’s question and the opportunity to clarify the motivation. AttEx-MoE improves generalization under domain shift by combining (1) geometry-aware, per-instance expert reweighting and (2) stable, soft channel modulation via Sigmoid gating, all within the PEFT setting. This design allows the model to adapt to instance-specific variations while preserving source knowledge through a frozen backbone.

Specifically, the row-wise self-attention router extracts informative epipolar-consistent context, enabling input-aware reweighting of channel experts without modifying the backbone. More importantly, Sigmoid-based gating provides smooth and bounded scaling of expert activations on a per-instance basis. This is critical, as these gating values are directly multiplied with expert channels (See Eq. (2)). In contrast to ReLU, which may cause hard-zeroing and unbounded scaling, Sigmoid gating ensures bounded and stable modulation, preserving gradient flow and improving robustness under distribution shift. The effectiveness of Sigmoid gating function is empirically supported in Tab. 4, where it consistently outperforms ReLU across all router variants.

Summary

• AttEx-MoE enhances generalization via input-dependent expert routing and stable Sigmoid-based gating.
• The frozen backbone preserves source knowledge, while the AttEx-MoE dynamically adapts to instance-specific features without architectural changes.

---

Weakness c. New Evaluation protocol

c-1. CTTA evaluation protocol

We acknowledge that Fig. 5 in [A] evaluates model performance in a looping scenario. However, it operates within a single domain (Alley 2 from Sintel [B]) where the target distribution is fixed. While useful for analyzing long-term adaptation, it does not assess adaptation under domain shift or knowledge preserving across diverse environments.

In contrast, our benchmark cycles through multiple, distinct domains (e.g., different weather conditions, nighttime scenes), introducing explicit distribution shifts. This allows us to evaluate continual adaptation challenges such as error accumulation and catastrophic forgetting, which fixed-domain setups cannot capture.

We will clarify this distinction and cite [A] in the revised manuscript.

c-2. TTA evaluation results

As noted by the reviewer, (j) RobIA shows smaller gains under the TTA benchmark (Tab. 9-11) compared to (i) CoEx-FULL++. This can happen as TTA operates within a single domain using longer sequences, favoring full-tuning methods with higher capacity.

This trend is also observed in MADNet 2, where (f) full tuning outperforms (g) modular update, particularly on KITTI ((f) 1.42%, (g) 1.64%) and DSEC ((f) 6.02%, (g) 7.44%), suggesting that high-capacity models benefit more from long-term adaptation in fixed-domain settings.

Nevertheless, (j) RobIA achieves comparable performance to full-tuned baselines (See Tab. 9, 11). Interestingly, in DrivingStereo (Tab. 10), where intra-domain variability is higher (e.g., dusky, cloudy, rainy), (j) RobIA (5.02%) outperforms (i) (5.2 %), indicating stronger robustness under dynamic conditions.

---

Weakness d. Comparison with Tonioni, Alessio, et al. 2019

[C] introduces two additional self-supervised losses---a smoothness loss $L_{smooth}$ and a photometric reconstruction loss $L_{photo}$---to compensate for sparse proxy labels via auxiliary supervision. This shares a similar motivation as our AdaptBN teacher.

To evaluate this strategy in our continual adaptation setting, we incorporated these loss terms into our framework as follows: $L$ = $L_{proxy}$ + $\lambda$ $\cdot$ $L_{smooth}$ + $\lambda$ $\cdot$ $L_{photo}$. In Table A, (a) and (e) corresponds to the baseline using only the proxy loss. (c) and (f) apply the above strategy with $\lambda=0.1$, but neither yields meaningful improvement over (a) and (d), respectively.

In contrast, (b) and (e)---our AdaptBN-Teacher loss---provides more reliable pseudo-labels, thus lowering the error ((a) 3.04% $\rightarrow$ (b) 2.93%, (d) 2.98% $\rightarrow$ (e) 2.77%). This result confirms the effectiveness of our supervision strategy in continual adaptation.

We will include this ablation in the revised manuscript and clarify that our teacher strategy outperforms simple photometric augmentation.

Table A

---

Question 1. Generation Time in Tab. 3

In the timing in Tab. 3, we included pseudo-label generation through AdaptBN teacher. However, we excluded SGM label generation time for the same comparison with existing TTA methods [D], which also used SGM as a proxy label but do not account for its generation time in their runtime analysis.

---

About Paper Formatting Concerns

We appreciate your careful reading and pointing out the formatting issues. We will correct the loss term and fix the boldface formatting in Table 1 and the other tables.

---

Citations

• [A] Tonioni, Alessio, et al. "Real-time self-adaptive deep stereo." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
• [B] Butler, Daniel J., et al. "A naturalistic open source movie for optical flow evaluation." European conference on computer vision. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012.
• [C] Tonioni, Alessio, et al. "Unsupervised domain adaptation for depth prediction from images." IEEE transactions on pattern analysis and machine intelligence 42.10 (2019): 2396-2409.
• [D] Poggi, Matteo, et al. "Continual adaptation for deep stereo." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.9 (2021): 4713-4729.