Q1: For small Weaknesses, "Imagenet: A large-scale hierarchical image database." appears twice in the references.

Thank you for your careful review. We will correct this in the final version.

Q2: How can authors effectively ensure that the generated images are truly positive or negative samples for generating tasks?

Thank you for your insightful question.

Our theoretical analysis ensures that the generated images are truly positive or negative samples. Furthermore, our experimental results as well as visualizations further verify this. The details are presented below:

• Theoretical analysis: Theorem 3.1 and 3.2 guarentee that our method can erase semantics in images for negative instance generation. For positive instances, we exploit a conditional diffusion model, starting from a seed sample-based initialization, to ensure the generated images resemble data distributions and exhibit the target semantics.
• Experimental results: If the generated images contained a high proportion of false positives or false negatives, the performance would be unsatisfactory. As shown in the ablation study in Figure 4(a), utilizing only the generated images to train the classifier improves performance, quantitatively confirming the correctness of the generated images.
• Visualizations: Section B.13 in the Appendix demonstrates that the generated images closely align with the expected characteristics of positive and negative samples, providing qualitative validation of the label correctness of the generated images.

Conclusion:

In the revision, we will include the above analysis in the conclusion of Section 3.4 as follows: "The theoretical analysis outlined in Theorems 3.1 and 3.2, along with the diffusion-driven approach for positive instance generation, confirms the reliability of both the generated negative and positive instances. This is further supported by the experimental results in Sec. 4.3 and the visualizations presented in Appendix B.13.".

Q3: Have the authors explored how to generate hard samples in the proposed method, namely those that are more helpful in improving model performance?

Thank you for your insightful question!

We incorporate hard samples into training through a confidence-based labeling module that identified difficult real instances, as shown in Section 3.5. These hard examples are then paired with generated images for training. By progressively lowering the confidence threshold, increasingly difficult examples are introduced during training. The results of our exploration of this approach are presented below:

Experimental results:

• The ablation study results (Figure 4(a)) show that integrating these hard training pairs effectively improves performance (pink bar), particularly when compared to training solely with generated images (green bar).
• As shown in Figure 4(c), incorporating an excessive number of harder samples (by lowering the confidence threshold) leads to unstable performance. This is because the pseudo-labels for hard samples are assigned based on the classifier's predictions. When too many hard samples are included, the risk of incorrect pseudo-labels increases, leading to performance instability.

Future exploration:

Thank you for the valuable suggestion. Your insight motivates us to further refine positive and negative instance generation to construct harder training pairs. This can be achieved by designing more specific prompts to erase critical features, improving the contrast between positive and negative samples.

We have already generated highly similar positive and negative pairs and provide examples in https://anonymous.4open.science/r/Rebuttal-UCDM-7787/Figure-Generated%20hard%20training%20pairs.pdf. We plan to further explore this direction in future work.

Q1: Rationale behind the proposed motivation: While positioned as an unsupervised method, the use of a conditional diffusion model to generate positive-negative instance pairs appears to implicitly incorporate label information through the class-conditional generation process. This creates potential ambiguity in maintaining true unsupervised learning principles, as conventional unsupervised approaches (e.g., self-supervised learning) typically evaluate through linear probing without explicit class guidance during representation learning.

Thank you for your insightful question.

Our problem and method align with unsupervised learning as defined in Deep Learning[1]. Additionally, class-conditional generation and explicit class guidance in unsupervised learning have been extensively explored in existing studies across various tasks.

• Definition of unsupervised learning: Our method trains the classifier with generated images and pseudo-labels without any human annotation, following Deep Learning[1], which defines unsupervised learning as "most attempts to extract information from a distribution that does not require human labor to annotate examples."

• With class-conditional generation: Our approach aligns with unsupervised domain adaptation[4,5], where class names and conditional diffusion models generate target data for training. This further confirms that introducing class-conditional generation is still considered unsupervised.

• With explicit class guidance in unsupervised learning: The "class name" setting in our method is commonly adopted in the unsupervised fine-tuning of multimodal models [2,3], where models are adapted to unlabeled target data, assuming target class names are known, but the mapping to the unlabeled data is not. Accordingly, this assumption is rational as well as widespread.

We summarize the commonalities between existing unsupervised studies and our work as follows.

Conclusion:

Our method adheres to the principles of unsupervised learning, as it does not rely on manually annotated labels.

We greatly appreciate your valuable feedback, we will supplement the following description in paragraph 4 of Section 1: "In this context, we aim to construct positive-negative pairs for training the classifier without any human annotation, adhering to the unsupervised learning setting [1].".

Reference:

[1] Deep learning. Cambridge: MIT press 2016.

[2] Pouf: Prompt-oriented unsupervised fine-tuning for large pre-trained models. ICML 2023.

[3] Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization. ICML 2024.

[4] One-shot unsupervised domain adaptation with personalized diffusion models. CVPR 2023.

[5] Domain-guided conditional diffusion model for unsupervised domain adaptation. Neural Networks 2025.

Q2: Figure clarity: Figure 3 is overly complex and difficult to follow; simplifying the training diagram would improve readability.

Thank you for your helpful suggestion.

Figure 3 illustrates the classifier training pipeline based on unlabeled data and generated instances. To facilitate readability, we have simplified the figure in the revised version, highlighting the following three stages:

• Stage 1: Generated positive and negative instances are used to create training pairs for classifier training, following Eq. (9).
• Stage 2: The trained classifier(frozen) selects confident real images from unlabeled data for pseudo-labeling, using Eq. (10) and Eq. (11).
• Stage 3: Training pairs are constructed using both selected and generated data, and the classifier is further trained following Eq. (12), similar to Stage 1.

The updated figure can be found at https://anonymous.4open.science/r/Rebuttal-UCDM-7787/Figure3-Simplified%20framework.pdf. Please feel free to check it.

Q1:Unvalidated assumptions of noise estimators.

Thanks for your question.

The assumption follows existing works[1,2,3]. We further verify its validity as follows.

• Same assumption in DDIM and DDIM inversion. DDIM[1] solves diffusion ODEs via forward Euler, where $\epsilon(\mathbf{x}\_t,t)\approx\epsilon(\mathbf{x}\_{t-1},t)$. DDIM inversion applies the same in reverse[2,3]. The approximation quality depends on $\mathbf{x}\_t-\mathbf{x}\_{t-1}$ and $\epsilon\_{\theta}$’s sensitivity to $\mathbf{x}\_t$[2].
• Validation. We analyze the discrepancy between $\epsilon(\mathbf{x}\_t, t)$ and $\epsilon(\mathbf{x}\_{t-1}, t)$, measured by 1-cosine similarity, over 20 DDIM steps. Results show near-perfect alignment, consistent with Fig.S10 in [3].
  step|1|2|3|4|5|6|7|8|9|10|11|...|14|15|...|19|20 -|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|- condition|4e-2|3e-2|2e-2|2e-2|1e-2|8e-3|5e-3|3e-3|3e-3|2e-3|2e-3|...|2e-3|1e-3|...|1e-3|4e-4 uncondition|4e-2|3e-2|2e-2|2e-2|1e-2|8e-3|5e-3|3e-3|3e-3|3e-3|2e-3|...|2e-3|1e-3|...|1e-3|4e-4

In the revision, we will supplement this validation to Appendix A.7.

Reference:

[1] Denoising Diffusion Implicit Models,ICLR 2021.

[2] Fixed-point inversion for text-to-image diffusion models,CoRR 2023.

[3] Edict: Exact diffusion inversion via coupled transformations,CVPR 2023.

Q2:Competitor models yield 0 precision on sub-tasks, raising model selection concerns.

Thanks for your question.

• Our method and all compared ones focus on class distribution mismatch(CDM), ensuring a fair and reasonable comparison.
• Two key factors explain the poor performance: method|reason for poor accuracy
  -|-
  $\text{DS}^3\text{L}$, UASD, CCSSL, T2T|Designed for closed-set task, unable to handle unknown or new classes
  MCTF, IOMatch, OpenMatch|Dependence on labeled data causes imbalanced accuracy across known, unknown, and new classes
• The Unsupervised CDM setting is newly proposed, and the performance gap highlights our method's strength in tackling this problem.

Q3: Small size of used datasets.

Thanks for your suggestion.

While we follow the same dataset evaluation as recent works, we have additionally evaluate on a larger dataset to further validate the effectiveness of our method, as recommended.

• Dataset selection: The datasets used are common in recent studies, ensuring fairness. ImageNet-30(39k images, 30 classes) used in some methods are smaller than Tiny-ImageNet(100k images, 200 classes) in our study. method|CIFAR10|CIFAR100|Tiny-ImageNet|Larger dataset -|-|-|-|- $\text{DS}^3\text{L}$|✅|❌|❌|❌ IOMatch|✅|✅|ImageNet-30|❌ OpenMatch|✅|✅|ImageNet-30|❌ MCTF|✅|❌|✅|❌ UASD|✅|✅|✅|❌ ours|✅|✅|✅|✅(below)
• Large-scale evaluation: We test on 763,577 images from a combination of CIFAR10, SVHN, Flower-102, and Food-101 datasets. Results show our method performs well on large datasets. method|close(acc.)|open(kno.)|open(unkno.)|open(new.)|open(bala.) -|-|-|-|-|- CCSSL|57.7|57.7|0|0|-14.1 T2T|60|60|0|0|-15.6 IOMatch|46.7|20.4|36.3|75.6|15.7 OpenMatch|19.8|17.7|18.9|12.5|13 ours|53|48.3|100|83.4|50.8

Q4:Small image sizes in experiments.

Thanks for your suggestion.

The image size follows prior works like MCTF and UASD. As recommended, we test on Tiny-ImageNet with $224 \times 224 \times 3$, and the results show similar trends to those in the paper:

Q5:Relation to domain generalization(DG) and its competitiveness.

Thanks for your suggestion.

We compare our setup with DG and open DG to clarify key differences and also compare it with a representative open DG method.

• Relation to DG. DG handles domain shifts (e.g., cartoon vs. natural images) without class mismatch. Open DG adds class mismatch but requires labeled data from multiple domains, while our setup is based entirely on unlabeled data. problem|unlabeled data|training set domains|train-test class mismatch|data shift -|-|-|-|- DG|❌|≥ 1|❌|✅ Open DG|❌|>1|✅|✅ ours|✅|1|✅|❌
• Comparison with open DG method DAML[1] on cross-dataset. Unlike ours, DAML is fully supervised with ground truth for all data, including unknown classes—an ideal but unrealistic scenario for unsupervised methods. Even so, our method achieves comparable accuracy on unknown and new classes, showing its effectiveness without labeled data. method|close(acc.)|open(kno.)|open(unkno.)|open(new.)|open(bala.) -|-|-|-|-|- DAML|81.5|62.3|99.9|85.5|63.6 ours|53|48.3|100|83.4|50.8

We appreciate this suggestion and will supplement a DG review and this experiment in the revised version.

Reference:

[1] Open domain generalization with domain-augmented meta-learning,CVPR 2021.