DAA: Amplifying Unknown Discrepancy for Test-Time Discovery

Q1: Effectiveness to Noise

Thank you for raising this important concern. To evaluate the robustness of our method under such conditions, we conducted a series of experiments introducing both label noise and image noise at a 10% corruption rate, under different stages (warm-up and test-time).

Despite the noise, our method consistently outperforms the baseline HM in both real-time and post evaluations. The results suggest that our design, especially the FIFO-based memory buffer and STMR’s error correction mechanism—provides substantial robustness and stabilizing effects under noisy conditions.

Q2: Insufficient benchmark

We appreciate the reviewer’s suggestion regarding the scale of evaluation benchmarks. While we agree that validating on larger or more diverse datasets is an important direction, we would like to emphasize that TTD is already a highly challenging task even on the current benchmarks. For instance, on Tiny-ImageNet-D—a dataset with 200 classes and significant visual diversity—discovering novel classes without any supervision remains extremely difficult. As shown in Figure 4, our method achieves only 24.10% average true-label agreement (TA), highlighting how hard it is to consistently cluster unseen class samples even with a well-trained backbone. This indicates that achieving reliable TTD performance on existing benchmarks is already non-trivial. Therefore, while scaling up is valuable, we believe that pushing the performance boundary on these already-challenging benchmarks is equally, if not more, important at the current stage. We appreciate the suggestion and plan to extend our evaluation to more complex datasets in future work.

Q3: Effectiveness of simulated samples

Thank you for the insightful question. To evaluate the effectiveness of our method under such conditions, we conducted a series of statistical analyses and comparative experiments.

Q3(a): maximum matching confidence statistic

Since our method discovers new classes based on maximum matching confidence, we analyzed the confidence scores of different sample types with respect to known-class prototypes. The results are summarized below:

These findings suggest that:

• Simulated samples occupy more ambiguous regions in the feature space, with lower confidence compared to known-class samples.
• Real unknowns discovered by our method are also harder (lower confidence) than those handled by HM, indicating better modeling of the decision boundary.
• Our method achieves a higher direct discovery success rate via thresholding, showing its superior ability to identify novel categories in ambiguous zones.

Q3(b): Effectiveness on samples lie outside known distributions

To examine our method’s behavior for unknown classes that lie far outside the known distribution, we introduced additional noise to these samples, making them more separable. Results are shown below:

The experiment shows that more distinct unknowns (i.e., farther from the known distribution) tend to have lower confidence and are easier to discover. This confirms that our method performs well in such cases even without simulation, as these samples are less ambiguous.

In summary, our simulation focuses on the most difficult unknowns—those near the known distribution, where class confusion is high. These are the scenarios where explicit modeling is most needed. Conversely, unknowns that lie far outside the known space are easier to identify and do not require sophisticated simulation to be discovered. This demonstrates the practical value and effectiveness of our simulated unknown training strategy.

Q4: Extend to Continual Learning scenarios

Our method can naturally generalizes to continual-learning settings. And to verify this compatibility, we conducted an experiment that follows the setting of continual stage in Continual Class Discovery (CCD) scenarios. We conducted the preliminary 3-stages experiment on Cifar-100 (70+10+10+10)

The results show that our framework is well-suited to continual class discovery, and these early results show promising potential for broader continual learning applications. Further exploration with more stages and longer task horizons is an exciting direction for future work.

Q5: Overall algorithm

To improve clarity, we provided an overall algorithm as follow.

Algorithm: DAA Training and Test-Time Discovery with STMR

Input:  
- Training set $ \mathcal{D}_{\text{train}} = \{(x_i, y_i)\}_{i=1}^{N_{\text{train}}} $ with known classes $ \mathcal{Y}_{\text{kn}} $  
- Test stream $ \mathcal{D}_{\text{test}} $ with unknown classes  
- Backbone model $ f $ (frozen), adapter $ \text{DAA} $ (trainable)  
- Hyperparameters: warm-up epochs $ E $, memory size $ M $, confidence threshold $ \gamma $  

Output:  
- Predictions $ \hat{y} $ for test samples, dynamically updated DAA and prototypes  


#Phase 1: Discrepancy-Amplifying Warm-up (Pre-Testing)

# Initialize DAA parameters θ_DAA
for epoch in range(E):
 for batch B in D_train:
   # Simulate unknown features via Mixup + Gaussian noise
   r = f(B)  # Backbone features
   r_tilde = mixup(r) + N(0, Σ)  # Synthetic unknowns
   
   # Compute losses
   L_kn = MSE(DAA(r), r)  # Preserve known semantics
   L_un = contrastive_loss(DAA(r), DAA(r_tilde))  # Amplify discrepancies
   L_train = L_kn + λ * L_un
   
   # Update θ_DAA via SGD
   θ_DAA ← θ_DAA - η * ∇L_train

#Phase 2: Test-Time Discovery with STMR

# Initialize memory queues
S_kn, F_kn = load_known_prototypes(D_train)  # Static for known classes
S_un, F_un = {}, {}  # FIFO queues for unknown classes (empty initially)

for batch B in D_test:
 # Extract features
 r = f(B)
 z = DAA(r)  # Adapted features
 
 # Prediction via prototype matching
 P = cosine_similarity(z, prototypes=[p_c for c in Y_kn ∪ Y_seen])
 y_hat = argmax_c P(c)
 
 # Confidence-based discovery
 if max(P) > γ:
   assign y_hat  # Known/seen class
 else:
   y_hat = "new_unknown"
   Y_seen.add(y_hat)
   initialize p_y_hat = mean(z)  # New prototype
 
 # STMR: Memory renewal
 if y_hat in Y_un:
   # Renewal step every T batches
   if batch_id % T == 0:
 for x in S_un[y_hat]:
   z_renew = DAA(f(x))
   if prediction(z_renew) ∈ Y_kn:
 discard x  # Remove misclassified known samples
   else:
 F_un[y_hat].enqueue(z_renew)  # Update features
   
   # FIFO update
   S_un[y_hat].enqueue(B)
   F_un[y_hat].enqueue(z)
 
 # Test-time training
 L_test = compute_loss(z, y_hat, S_un, F_un)  # Eq. (6) in paper
 θ_DAA ← θ_DAA - η * ∇L_test  # Self-supervised update

Thank you again for your time and effort in reviewing our submission. We have submitted our response to your questions and would appreciate it if you could take a moment to review our response. Please feel free to let us know if any further clarification is needed.

Q1: Architecture of DAA

The DAA adopts a standard 2-layer adapter [1] inserted after the frozen ViT-B/16 backbone. It consists of:

• Linear down-projection: 768 → 128
• ReLU activation
• Linear up-projection: 128 → 768

This design introduces minimal overhead (∼200K parameters). We emphasize that the adapter structure itself is not our contribution. It follows a widely used design. Our key innovation lies in the training strategy that leverages this lightweight adapter to retain known-class features and amplify discrepancies for unknowns, enabling effective test-time discovery. We will add the architectural details to the appendix for clarity.

[1]Houlsby, Neil, et al. "Parameter-efficient transfer learning for NLP." International conference on machine learning. PMLR, 2019.

Q2: Advice for Contribution

Thank you for the suggestion. We agree that the core contributions lie in DAA and the Short-Term Memory Renewal (STMR), and we will revise the introduction to reflect this more clearly. The reported performance gains serve as empirical validation of these two design components, rather than standalone contributions. We appreciate the feedback and will streamline the contribution statement accordingly.

Q3: Robustness to Noise

Thank you for raising this important concern. To evaluate the robustness of our method under such conditions, we conducted a series of experiments introducing both label noise and image noise at a 10% corruption rate, under different stages (warm-up and test-time).

Despite the noise, our method consistently outperforms the baseline HM in both real-time and post evaluations. The results suggest that our design, especially the FIFO-based memory buffer and STMR’s error correction mechanism—provides substantial robustness and stabilizing effects under noisy conditions.

Q4: Hyperparameters

Q4(a): Similarity threshold $\gamma$

$\gamma$ was selected through grid search in our experiment, and we analyzed the impact of $\gamma$ on a single dataset in the Appendix. The typical threshold of $\gamma$ is 0.7. And our method is not sensitive to selection from one dataset to other datasets. We conducted the following experiments:

Q4(b): Memory size

We have analyzed different memory size in terms of agreement and time consumption in Figure 6 in the manuscript. And the typical size of the memory queue is 10. While larger memory can store more samples, it may include outdated or irrelevant data, weakening prototype quality. Conversely, too small a memory may fail to capture representative features, reducing generalization. Additionally, increasing memory size leads to higher computational cost. We also analyze the size choice around 10. The results indicate that small differences around 10 have little impact on the effectiveness.

Q4(c): Warm-up duration

Few warm-up epochs result in insufficient effectiveness, but when the number of rounds reaches a certain level, the effect tends to converge and the improvement is limited.

Q4(d): Loss weights

The weight of two loss during warm-up phase is fixed (200:1 in paper), and we also tried other combinations as shown in the table below.

During testing phase, we have discussed the influence of weight in the appendix, and we also show the resulst in the table below.

Q4(e): Mixup beta parameters

In our implementation of the Mixup strategy, we conducted Symmetric Beta Distribution, and the default Beta parameters is 0.5. And we also conduct the hyperparameter sensitivity analysis.

The small difference in Beta parameters of Mixup around 0.5 leads to a similar effect in final discrimination ability. Mixing situations that are too high or too low can result in poor simulation performance

Q5(a): Why was FIFO

Thank you for the question. In our design: The FIFO is used in the discovery phase and the random sampling is used in the correction phase. The sample queue can accumulate knowledge of new classes in a timely manner during the discovery phase, but it may introduce errors, so the correction phase uses random sampling for correction. To ensure efficiency, we have found that random sampling is already good and may consider better solutions in the future, such as selecting samples with higher uncertainty.

Due to the fact that the confidence level of new samples may be low when discovered. As the testing phase progresses, the samples with low confidence level at the time of discovery may have actually become higher.

Q5(b): Frenquency of STMR

Thank you for the question. We treat the STMR renewal frequency as a tunable hyperparameter, and have reported a comparative analysis in the Appendix. In our main experiments, we set the renewal frequency to 2 batches, which provides a good trade-off between performance and efficiency:

A lower frequency of STMR leads to insufficient updates and refinements of the model during the test phase, resulting in poorer recognition performance for both known and novel classes.While increasing the frequency of STMR can improve the model's ability to adapt to new data, it also leads to a substantial increase in computational overhead. Frequent STMR operations require more time and resources for processing each batch. We currently use a fixed schedule, but exploring adaptive strategies based on confidence or drift detection is a promising direction for future work.

Thank you again for your time and effort in reviewing our submission. We have submitted our response to your questions and would appreciate it if you could take a moment to review our response. Please feel free to let us know if any further clarification is needed.

Dear reviewer, please engage with the author rebuttal.

Thank you very much for raising these important review points.

Weakness

W1

Due to space constraints, we have omitted some parts. We will rearrange the article and add some descriptions in the main text to make it clearer.

• sim() denotes cosine similarity between two vectors: $\text{sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \|\mathbf{b}\|}$

• About Eq.2, we further modified the equation to avoid confusion and increase readability: $\mathcal{L}\_ {\text{un}}(x) = -\log \frac{\exp(\sigma[\text{DAA}(\mathbf{\tilde{r}}(x)),\text{DAA}(\mathbf{\tilde{r}}(x))\_ {aug}])}{\sum\_ {x' \in \mathcal{B}} \exp(\sigma[\text{DAA}(\mathbf{r}(x')), \text{DAA}(\mathbf{\tilde{r}}(x'))])}$, where x' iterates over all samples in the mini-batch $\mathcal{B}$ including known classes samples and simulated unknown samples. $\text{DAA}(\mathbf{\tilde{r}}(x))\_{aug}$ is the feature obtained through data augmentation operations.

• Captions for figures: We will expand figure captions to clarify axes, units, and color meanings. For example, in Fig. 4: x-axis = methods, y-axis = agreement (%), purple = TA, red = CA.

• In Equation (7), $\mathcal{L}\_ {\text{kn}}$ and $\mathcal{L}\_ {\text{un}}$ represents batch level calculation.

• “real-time” eval and “post-eval”: We follow HM and define both clearly. Real-time eval reflects online performance during streaming; post-eval re-evaluates all samples after testing completes.

W2: Implementation details

Thank you for pointing out the reproducibility-related concerns. We address each part below and will clarify these points in the main text or appendix.

W2(a): The choice of $\gamma$

$\gamma$ was selected through grid search in our experiment, and we analyzed the impact of $\gamma$ on a single dataset in the Appendix. The typical threshold of $\gamma$ is 0.7. And our method is not sensitive to selection from one dataset to other datasets. We conducted the following experiments:

W2(b) Weight of two loss in Eq.(3)

The weight of two loss during warm-up phase is fixed (200:1 in paper). We will add the weight parameters to avoid assuming a 1:1 ratio, and we will include different comparison results as below. We have provieded different ratio of weight parameters during warm-up in the appendix, and we also show it as follows.

During testing phase, we have discussed the influence of weight in appendix. And the results is also shown in the following table.

During both warm-up and testing phase, the MSE loss is used to refine the model's predictions for known classes, while the contrastive loss encourages the model to separate the feature embeddings of known and unknown classes. When the model places a higher emphasis on known classes, the model's ability to distinguish between known and unknown classes is limited. As the ratio decreases, the model starts to pay more attention to the contrastive loss, which helps improve the separation between known and unknown classes.

W2(c) Architecture of DAA

The DAA adopts a standard 2-layer adapter [1] inserted after the frozen ViT-B/16 backbone. It consists of:

• Linear down-projection: 768 → 128
• ReLU activation
• Linear up-projection: 128 → 768

This design introduces minimal overhead (∼200K parameters). We emphasize that the adapter structure itself is not our contribution. It follows a widely used design. Our key innovation lies in the training strategy that leverages this lightweight adapter to retain known-class features and amplify discrepancies for unknowns, enabling effective test-time discovery. We will add the architectural details to the appendix for clarity.

[1]Houlsby, Neil, et al. "Parameter-efficient transfer learning for NLP." International conference on machine learning. PMLR, 2019.

Q2(d) Typical size of the memory queue

The typical size of the memory queue is 10 as we analyze different memory size in terms of agreement and time consumption in Figure 6 in the main paper. While larger memory can store more samples, it may include outdated or irrelevant data, weakening prototype quality. Conversely, too small a memory may fail to capture representative features, reducing generalization. Additionally, increasing memory size leads to higher computational cost.

W3(a): Do x and x' come from two different categories

The samples x are from the set of known classes and x' are created by Mixup-Gaussian augmentation on other samples in the batch. These simulation method is widely used in many work, such as OpenMix [2] and HiLo [3]. Although the mixture do not produce a real unknown category, they can effectively simulate unknown classes.

We conducted experiments and the results showed that the average confidence of simulated samples was lower than that of known class samples (lower confidence is easier to be discovered); The average confidence level of the truly unknown samples obtained by our method is also lower than that of the unknown samples under the HM method. Similarly, our method has a higher success rate than the HM method in directly discovering new classes through threshold method.

[2]Zhong, Zhun, et al. "Openmix: Reviving known knowledge for discovering novel visual categories in an open world." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[3]Wang, Hongjun, Sagar Vaze, and Kai Han. "Hilo: A learning framework for generalized category discovery robust to domain shifts." arXiv preprint arXiv:2408.04591 (2024).

W3(b): For Eq 2, why is MSE used and how contrastive loss work

The MSE keeps the adapter output close to the original backbone feature when the input is a known sample. This acts as a regularizer that prevents forgetting and destruction of the known classes. Known classes features and simulated features as a mini-batch, simulated features and feature augmented simulated features as positive samples, and simulated features and all other data within the small batch as negative samples.

W4: Frenquency of STMR

We made the comparison about frenquency of STMR as a Hyper-parameter analysis in Appendix. The typical number of frequency in paper is 2.

A lower frequency of STMR leads to insufficient updates and refinements of the model during the test phase, resulting in poorer recognition performance for both known and novel classes. While increasing the frequency of STMR can improve the model's ability to adapt to new data, it also leads to a substantial increase in computational overhead. Frequent STMR operations require more time and resources for processing each batch.

Question & Limitation:

Q1: Checklist

This is a typographical error, and our submission does not contain formal theorems or explicit theoretical assumptions. However, the claims regarding the effectiveness of DAA and STMR are supported by extensive experimental validation across multiple datasets and ablations. For example, Fig. 5 in the main paper and the t-SNE visualization in the main paper, and the appendix validate our effect.

Q2: Computational cost

In terms of time cost, we conducted an experiment on Cifar-100 (70+30) as shown in the following table.

The results show that our time consumption is relatively small. DAA is efficient: Compared to GMP, which updates the full backbone and incurs heavy overhead, and HM, which performs test-time neighbor search, our approach is both faster and lighter. PHE is faster, but it lacks any test-time adaptation and thus sacrifices performance. Our memory design uses a short FIFO queue, which is more compact than HM’s bucket/hash-based storage. In conclusion, our method achieves strong performance with only minor overhead, making it practical for real-world usage.

Thank you again for your time and effort in reviewing our submission. We have submitted our response to your questions and would appreciate it if you could take a moment to review our response. Please feel free to let us know if any further clarification is needed.

Dear reviewer, please engage with the author rebuttal.

Dear Reviewer,

Thank you for your review. As we approach the end of the Author–Reviewer discussion period, we want to follow up and confirm whether our response was able to address your concerns. If you feel that the clarifications were satisfactory, we would be sincerely grateful if you would consider updating your score accordingly. If there are any remaining questions you’d like us to further clarify, we’d be more than happy to provide additional information.

Best,

The Authors

Thank you very much for raising these important review points.

Q1: If replay memory stores known samples from the source dataset during test time

We agree that the strictest formulation of test-time training typically prohibits access to source data. However, prior works in both test-time adaptation and TTD often relax this constraint for the purpose of evaluating knowledge retention and stability. For example, methods such as CoTTA [1] and RoTTA [2] utilize source data during adaptation, and the previous HM method for TTD also maintains known-class samples throughout the test phase. Our usage of stored known samples in DAA’s Short-Term Memory (STMR) aligns with this precedent, aiming to mitigate forgetting of known classes.

To directly address the reviewer’s concern, we conducted an ablation study comparing different initialization strategies for the known-class memory, including settings that exclude training data entirely. The results are summarized below:

We observe that:

• Even when the memory is initialized as empty and constructed entirely from test-time samples (with or without updating), our method still maintains competitive performance.
• Using training-set-based initialization yields slightly better performance and lower forgetting, confirming its stabilizing effect on continual adaptation.
• Importantly, the performance drop in the test-only setting is relatively modest, indicating that our method is robust even without access to training data at test time.

These results reinforce that while using source-known samples can further enhance performance, our method remains effective under stricter constraints.

[1]Wang, Qin, et al. "Continual test-time domain adaptation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2]Yuan, Longhui, Binhui Xie, and Shuang Li. "Robust test-time adaptation in dynamic scenarios." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Q2: Implementation details of DAA

Thank you for pointing this out. DAA is explicitly designed to both preserve known-class representations and amplify feature-level discrepancies for unknown classes through tailored training strategies applied to a lightweight adapter layer.

• For preserving known-class features, we apply a mean squared error (MSE) loss between the backbone features and the DAA-adapted features, ensuring minimal distortion for known representations.
• For amplifying inter-class discrepancy among unknown classes, we employ a contrastive loss on the adapted features of simulated or discovered unknown samples, encouraging separation in the feature space.

During test time, the same dual-loss strategy is applied immediately after prediction to enable continual adaptation. Moreover, the Short-Term Memory Renewal (STMR) mechanism enhances this process by selectively replaying known samples and renewing unknown prototypes, which further guides DAA’s online updates and maintains discriminative capacity throughout the test stream.

Q3: Eq.(2) is unclear

We further modified the equation to avoid confusion and increase readability. The Eq.(2) was modified as follows:

Here, x' iterates over all samples in the mini-batch $\mathcal{B}$ including known classes samples and simulated unknown samples. $\text{DAA}(\mathbf{\tilde{r}}(x))_ {aug}$ is the feature obtained through data augmentation operations.

Q3(a): Class labels for synthesized unknown samples

Due to the contrastive loss only needs to know whether they belong to the same category or different. For example, $\text{DAA}(\mathbf{\tilde{r}}(x))$ and $\text{DAA}(\mathbf{\tilde{r}}(x))_{aug}$ are simulated samples and data augmented simulated samples, which are considered the same label. Different samples in the mini-batch $\mathcal{B}$ are considered different.

Q3(b) Weight of two loss in Eq.(3)

We will add the weight parameters to avoid assuming a 1:1 ratio, and we will include different comparison results as below in the appendix. Due to the significant difference in the magnitude of the two losses, if a 1:1 ratio is used, the model will collapse. The weight of two loss during warm-up phase is fixed (200:1 in paper)

Q4: The memory update mechanism in STMR

The Short-Term Memory mechanism involves two distinct phases with different sample selection strategies:

• During the discovery phase, we adopt a simple FIFO policy to quickly accumulate samples of newly discovered unknown classes. This enables the memory to capture temporal knowledge in real time, facilitating prototype initialization and early-stage adaptation.
• During the correction (renewal) phase, we use random sampling from the memory to update prototypes and adapt DAA. This strategy avoids bias toward any specific samples and supports stable online learning.

We chose random sampling in the correction phase primarily for efficiency and robustness. As shown below, the average confidence of unknown samples improves over time, suggesting that early uncertainty does not necessarily reflect long-term reliability:

This indicates that filtering based on early confidence scores could be both ineffective and computationally expensive, as low-confidence samples may become more reliable later. While more sophisticated filtering strategies (e.g., confidence-based selection or uncertainty estimation) are promising directions, our current design prioritizes simplicity and has proven effective in practice. We leave more advanced selection policies for future exploration.

Thank you again for your time and effort in reviewing our submission. We have submitted our response to your questions and would appreciate it if you could take a moment to review our response. Please feel free to let us know if any further clarification is needed.

Dear reviewer, please engage with the author rebuttal.