Response to Reviewer hVAv

We are grateful for your time and insightful comments!

W1 & Q1 - Generalizability beyond SparseDrive: explanation of the underlying causes of degradation, clarification of whether the limitation lies in the TTA paradigm or CodeMerge itself, and empirical evidence or arguments demonstrating broader applicability beyond SparseDrive.

Early end‑to‑end models under corruption. Our experiments show that some early architectures (e.g., VAD) collapse on corrupted scenes even before any TTA is applied. For example, under heavy snow, VAD attains only 0.0168 mAP for detection and 0.0353 NDS, indicating that the base model’s predictions are already unreliable and model almost collapsed. In this regime, online self‑supervision becomes untrustworthy, so any TTA method has little signal to learn from. Importantly, this reflects a robustness limitation of the base model, not of TTA or CodeMerge. Even so, attaching CodeMerge to VAD yields small but consistent gains across perception and planning metrics, and does not degrade VAD’s performance, which underscores that the bottleneck is the model’s severe brittleness rather than our adaptation mechanism.

Recent architectures with non‑collapsed baselines. When the backbone remains minimally competent under corruption, CodeMerge is highly effective and broadly applicable. On DiffusionDrive [A] and MomAD [B] (both in CVPR 2025), CodeMerge delivers consistent improvements across detection, tracking, mapping, motion, and planning, and it reduces collision rate under both Brightness and Snow corruptions. These results support that CodeMerge is able to scale beyond SparseDrive.

[A] Liao, etc. Truncated Diffusion Model for End-to-End Autonomous Driving. CVPR2025

[B] Song, etc. Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Driving. CVPR 2025

Q2 - Does this "frozen feature extractor" strategy also apply to the LiDAR-only detector experiments on SECOND, or are more parameters adapted and merged in that setting?

For 3D detection, prior works [C, D, E] have shown that adapting all weights of the entire detector during TTA yields the best performance, and we simply adopt that established experimental setup. Therefore, we train entire networks and then adapt and merge all parameters. Please also refer to Reviewer G337 W3.

[C] Chen, etc. MOS: Model Synergy for Test-Time Adaptation on LiDAR-Based 3D Object Detection. ICLR 2025

[D] Chen, etc. DPO: Dual-Perturbation Optimization for Test-time Adaptation in 3D Object Detection. ACMM 2024

[E] Yuan, etc. Reg-TTA3D: Better Regression Makes Better Test-Time Adaptive 3D Object Detection. ECCV 2024

Response to Reviewer G337

Thanks for your constructive and valuable feedback, which will greatly assist in enhancing our work!

W1 - Back-propagate through buffer? How many steps per frame?

No. All stored checkpoints are frozen. We never back‑propagate through past models. At step $t$, we (i) compute merging weights in fingerprint space and form a sign‑consistent merged teacher $\bar{\Theta}^{(t)}$ from the top‑$K$ buffered checkpoints, and (ii) use $\bar{\Theta}^{(t)}$ only to generate pseudo‑labels for the incoming batch. Only the current model $\Theta^{(t)}$ is updated once; other checkpoints are used for merging without gradient updates. This avoids the $K$-forward‑pass overhead of MOS [A], which is why our runtime and memory are lower (refer to Table 6 in the paper and response to W1 of reviewer jES4).

W2 - Robustness under fog + LiDAR dropout

We conduct additional experiments adapting SECOND from KITTI to KITTI-C under heavy fog with increasing point-dropout (0%→25%→50%) and report the AP 3D and AP BEV at the moderate level in the table below.

With increasing point-dropout, performance degrades gradually rather than catastrophically: AP 3D drops from 75.96→73.87→71.95 (–4.01 %, ≈5.3% relative), and AP BEV from 88.16→85.73→85.04 (–3.12 %, ≈3.5% relative). This smooth, monotonic decline suggests that our checkpoint ranking/merging remains reliable even under challenging compounded shifts.

W3 - Does CodeMerge still work when the backbone is adapted?

E2E model: In the paper’s end‑to‑end setting, we froze all components except the 3D box head to isolate the effect of merging and keep TTA lightweight. We conduct additional experiments where feature extraction backbone (ResNet) is also adapted and observe negligible deltas relative to head‑only TTA (e.g., NDS −0.0034, mAP −0.0055, AMOTA −0.0028, map mAP −0.0097, L2‑Avg −0.0023), indicating that CodeMerge remains effective when the backbone itself is updated. Furthermore, we have evaluated the effectiveness of CodeMerge by changing to different backbone models (see Reviewer hVAv, W1 & Q1), demonstrating the method’s broad applicability.

3D detection model: We simply follow prior works [A, B] to adapt the entire backbone (encoder, detection head, and all batch‑norm layers) as these works have already shown that, for test-time adaptation in 3D detection, adapting the full backbone leads to superior performance than adapting batch norm layers only.

[A] Chen, etc. MOS: Model Synergy for Test-Time Adaptation on LiDAR-Based 3D Object Detection. ICLR 2025

[B] Chen, etc. DPO: Dual-Perturbation Optimization for Test-time Adaptation in 3D Object Detection. ACMM 2024

W4 — Offline UDA (SN, ST3D) vs. online TTA (CodeMerge): pros & cons

Different Assumptions and Settings

UDA (SN, ST3D): has offline access to unlabeled target data and retrains the model for multiple epochs with self‑training / distribution alignment (e.g., ST3D), or target‑size statistics (SN). This typically yields strong target‑specific models when the target distribution is relatively stationary, but incurs heavy computation and latency before deployment.

TTA (CodeMerge): adapts online at inference on a stream of target frames. There is no multi‑epoch retraining. So each step is low-cost and fits real-time constraints.

Strengths & trade‑offs

UDA (SN, ST3D):

• Pros: Can fully specialize to the target dataset with extensive training; strong performance when the deployment domain is fixed.
• Cons: Requires offline target data and multi‑epoch computation; not reactive to rapidly changing in‑field dynamic conditions.

TTA (CodeMerge):

• Pros: Immediate, frame‑by‑frame adaptation suited to dynamic shifts; efficient (e.g., vs. MOS, –41.8% runtime and –8% memory on SECOND; even larger savings on SparseDrive). Robust improvements are observed on cross‑dataset and corruption benchmarks, often surpassing UDA baselines (e.g., > ST3D on nuScenes→KITTI).
• Cons: No offline access to target data; improvements depend on on‑the‑fly pseudo‑labels and the diversity/quality of the checkpoint buffer, so substantial shifts may require several steps to accumulate useful checkpoints. (We mitigate this with geometry‑preserving fingerprints and curvature‑aware selection.)

W5 - Merging baseline: EMA

We conducted additional experiments by replacing the merged model with the EMA model. On the same setting in Table 5 of the paper, CodeMerge (k=5, proj‑dim=1024) consistently outperforms EMA across all tasks: Detection mAP/NDS +0.0373/+0.0272; Tracking AMOTA +0.0372 and AMOTP −0.0810 (↓ better); Mapping mAP/APped +0.0355/+0.0300; Motion mADE/mFDE −0.0147/−0.0301; Planning L2‑Avg/CR‑Avg −0.0274/−0.0160. These gains indicate that our fingerprint‑guided merging, which selects complementary checkpoints, offers stronger generalization than the uniform, time‑local smoothing performed by EMA.

Thanks again for your helpful comments and questions!

Response to Reviewer jES4

W1 - Computational cost of each TTA step and comparison with baselines on SECOND

Thank you for your valuable comments! The runtime in Table 6 of our paper is the total time aggregated over all adaptation steps across the full test set. Below we report GPU memory and per-frame latency with AP 3D for SECOND on nuScenes→KITTI.

Speed comparison. Compared with MOS [A], CodeMerge cuts latency by ~43% (0.49→0.28 s) and raises AP 3D by +14.5%. Relative to TENT, +210.9% AP 3D at similar latency (0.26 vs. 0.28 s). Compared with CoTTA, only +0.13 s/frame yet +22.9% AP 3D.

Memory comparison. CodeMerge uses less memory than MOS (16,041 vs. 17,411 MiB) and slightly more than CoTTA (15,099 MiB), while achieving the highest AP 3D. Compared with TENT, it needs +5,209 MiB (~5.1 GiB) but keeps similar runtime and delivers significant AP gain: +210.9%.

Overall, CodeMerge offers the best accuracy–efficiency trade-off on SECOND for nuScenes→KITTI: highest AP 3D with competitive latency and memory, suitable for real-time TTA.

W2.1 - Rationale behind random projections.

Efficiency. Random projection primarily improves memory and runtime. We use a fixed Gaussian projection to compress intermediate features into compact “fingerprints” (262,144→1,024), stored in a codebook to compute merge weights. This training-free step keeps memory/compute proportional to fingerprint dimension (not full model size) and avoids loading/forwarding K past models (as MOS), yielding markedly lower latency and memory in practice (e.g., –41.8% runtime and –8% memory vs. MOS on SECOND).

Fidelity. Despite compression, the fingerprint space preserves model-space geometry [B]; pairwise fingerprint differences correlate strongly with parameter differences across architectures/datasets (Pearson 𝑟, Kendall τ>0.7). Our ridge-leverage scoring in fingerprint space links to inverse curvature in parameter space, justifying that these low-dimensional features carry the information needed for reliable checkpoint selection.

Robustness. Ablations (Table 5 of our paper) over projection dimension ($d' \in {256,512,1024,2048}$) show small variation, with $d'=1024$ the best accuracy–efficiency trade-off.

In summary, random projections give a simple, training-free, compute-efficient way to summarize checkpoints while retaining the structure needed for stable, effective test-time merging.

[A] Chen, etc. MOS: Model Synergy for Test-Time Adaptation on LiDAR-Based 3D Object Detection. ICLR 2025

[B] GONG, etc. Random Projections and Pre-trained Models for Continual Learning. NeurlPS 2023

W2.2 - Redefining projections at each time step?

No. We instantiate a fixed Gaussian random projection once before TTA and keep it frozen. It is applied to the pretrained source feature extractor’s intermediate activations $\phi_{\Theta^{(0)}}(x)$ to produce low-dimensional fingerprints $\hat z$ (Eq. 7 of our paper), embedding all checkpoints into a shared space across time. Because both the fingerprinting features (from the frozen extractor) and the projection are fixed, this mapping is training-free and lightweight.

W2.3 - Does the inherent randomness introduce performance variance?

We verify robustness by resampling the projection with three seeds (RP1–RP3) and re-running the full pipeline. The results vary slightly: Perception NDS = 0.4920–0.4946, mAP = 0.3630–0.3735; Tracking AMOTA = 0.3252–0.3347; Planning L2-Avg = 0.6209–0.6255. These small ranges indicate minor impact on end-to-end performance.

This stability is expected because (i) the projection is a fixed Gaussian linear map instantiated once and kept frozen during TTA, applied to features from the fixed source encoder to produce low-dimensional fingerprints (Eq. 7 of our paper), embedding all checkpoints in a shared space; (ii) the fingerprint space faithfully mirrors parameter-space geometry (Figure 3 of our paper: Pearson $r$ and Kendall $\tau$ typically $>0.7$), so random draws that preserve this embedding yield similar merge decisions.

Overall, repeated runs with different seeds show only minor fluctuations, consistent with our design for a stable, geometry-preserving random projection.

Q1 - Alternative fingerprint generation strategies?

Start with lightweight strategies. We thank you for your constructive suggestions. We have incorporated efficient feature aggregations: MaxPool [C], AdaptiveAvgPool [D], Lp-Pooling [E], and Fractional-Max [F]. Although inexpensive, these compress features to coarse statistics and underperform on Waymo→KITTI; our random projection (RP) achieves $AP_{3D}=63.23$ vs. 58.58/60.34/61.40/61.78 for pooling (gains +4.6%, +2.88%, +1.82%, +1.44%). This suggests simple pooling loses the discriminative structure needed for behavior-aware merging.

[C] Scherer, etc. Evaluation of pooling operations in convolutional architectures for object recognition. ICANN 2010

[D] He, etc. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. ECCV 2014

[E] Boureau, etc. A theoretical analysis of feature pooling in visual recognition. ICML 2010

[F] Zhai, etc. Pooling With Stochastic Spatial Sampling. CVPR 2017

Heavier dimensionality reduction isn’t practical online. Methods like PCA require fitting (or updating) decompositions on streaming features, adding substantial compute at every TTA step, which is incompatible with our single-pass, real-time constraint.

In summary, pooling is fast but loses geometry/accuracy; PCA-like reductions are too costly per step. A fixed random projection preserves the geometry needed for robust merging while meeting real-time TTA efficiency, and empirically delivers the strongest performance in our comparisons.

Q2 - Applicability beyond SECOND

To assess generality, we evaluate CodeMerge on the recent 3D detector DSVT [G] for Waymo → KITTI. CodeMerge reaches 79.88 AP BEV / 61.06 AP 3D, outperforming MOS by 3.2% and 6.4%, and surpassing No Adapt by 125% in AP 3D. Because CodeMerge builds on training-free random-projection fingerprints and codebook-guided merging rather than architecture-specific modules, these results indicate our TTA is model-agnostic and transfers to detectors beyond SECOND.

[G] Wang, etc. DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets. CVPR 2023

Q3 - Can fingerprints (from a fixed encoder on input $x_i$) reliably key the model parameters?

Yes. Empirically, pairwise distances in fingerprint space track those in weight space across datasets and models (Figure 3), with Pearson $r$ and Kendall $\tau$ typically >0.7, indicating that the low-dimensional geometry preserves parameter-space structure and supports reliable merge decisions.

Theoretically, under a ridge-regression surrogate for the linear detection head (with a fixed encoder), the ridge leverage score of a fingerprint is proportional to the directional inverse curvature $z_i^\top H_w^{-1} z_i$, so computations in fingerprint space provide a curvature-aware proxy for selecting informative parameter directions (Eq. 9–10). This provides a principled basis for using fingerprints as keys to the parameters they represent.

Finally, resampling the projection with different seeds (refer to W2.3) yields minor performance variation, further confirming that fingerprinting is stable and robust to projection randomness.

Once again, we sincerely appreciate your thoughtful review and constructive feedback! We will carefully incorporate these discussions into the revised manuscript.

Response to Reviewer XHzU

W1 - Code availability and pretrained models.

Code availability. Thank you for pointing this out. The source code was included in the Supplementary Material as a ZIP archive attached to the submission.

Pretrained models. The pretrained and adapted checkpoints for the end‑to‑end models and the 3D detectors are not uploaded with the submission due to file‑size limits. We will release all checkpoints publicly upon acceptance together with a reproducible repository that includes experiment configs, logs, and instructions for all experiments.