OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning

We deeply appreciate your time and effort in providing us with such constructive comments. We would like to respond to them as follows:

Q1: Generalization to truly unseen domains, e.g., cross-dataset evaluation.

Following your insightful suggestion, we extend our evaluation to investigate the generalization performance of our optimization framework in more challenging cross-dataset open-world scenarios. The experimental settings are listed as follows:

• We train our prompt exclusively on the ImageNet-500 base domain and subsequently evaluate model performance on a combined test set containing both the original ImageNet-500 classes and new categories from seven external datasets: FGVC-Aircraft, Caltech-101, Stanford-Cars, DTD, EuroSAT, SUN397, and UCF101.
• To ensure a fair evaluation of open-world generalization, we meticulously remove any categories from these external datasets that overlapped with the ImageNet-500 class space before evaluation.

The comprehensive results of this cross-domain evaluation, which rigorously tests the model's ability to handle both known and new categories across diverse visual domains, are presented in table below. The experimental results further speak to the efficiency of our method.

Q2: The trade-offs between performance and complexity of mixture-of-prompts.

Thank you for your constructive suggestion! In fact, we have included the prompt complexity analysis in Fig.5 in the initial submission. To highlight the efficiency of our method, we present those numerical results in the table below, along with additional results on inference speed.

• The table compares average performance on three open-world tasks and learnable parameter counts (#param) across methods

• The table presents a comparison of per-sample inference time, averaged across ten datasets

According to these empirical results, we answer the questions raised by the reviewer:

• Our method outperforms SOTA methods on the average performance of three open-world tasks with a smaller parameter cost. While recent SOTA methods design deep prompt structures, we optimize multiple shallow prompts in the detector and classifiers.

• While slightly slower than CoOp due to prompt mixing, our approach runs 34% faster than DeCoOp and matches DePT/Gallop in speed, which maintains practical inference speeds.

• The performance gain outweighs complexity. Our method outperforms SOTA methods by 1.94% –13.55% across tasks while using ≤ 9% parameters of methods like Maple or DePT. The moderate speed trade-off (slightly slower than CoOp) is justified by significant performance improvements.

Q3: Fairness-sensitive real-world scenarios application of the proposed method

The cross-dataset task mentioned above involves a significant imbalance between the base and new domains, as shown in the table below. The base domain includes 500 ImageNet categories with 25,000 testing samples.

Accuracy-driven metrics often favor the majority domain of ImageNet, creating fairness risks in scenarios with extreme data imbalances (e.g., 15× sample or 50× class ratios). These imbalances mirror real-world cases where critical minority samples are suppressed by dominant base classes. OpenworldAUC overcomes this bias by fairly evaluating all classes through pairwise ranking. Unlike traditional metrics that amplify imbalance-related errors, OpenworldAUC improves reliability in critical applications.

We deeply appreciate your time and effort in providing us with such constructive comments. We would like to respond to them as follows:

Q1: More detailed explanation of the challenges associated with generalization analysis.

• Our main theoretical findings focus on how well our optimization method generalizes to unseen test data distributions with new classes. According to standard learning theory, we measure generalization error by comparing: (a) the expected error across the joint distribution of the real new domain $\mathcal{Y}$ and test data $\mathcal{D}$, and (b) the empirical average across the pseudo new domain $\hat{\mathcal{Y}}$ and training data $\mathcal{S}$. To construct the pseudo new domain, we perform $K$ pseudo base-to-new partitions $\hat{\mathcal{Y}}^{(k)}, k \in [1, K]$, which incorporates a hierarchical sampling approach, class sampling, and data sampling. This leads to a hierarchical structure of stochastic errors, presenting a significant challenge in our theoretical analysis.
• The another major challenge arises from the AUC-type risk $\ell_{sq}(r(x_b;\theta_r) - r(x_n;\theta_r))$ in optimizing the detector $r$. The standard generalization analysis techniques, such as Rademacher complexity-based theoretical arguments[1,2], require the loss function to be expressed as the sum of independent terms. Unfortunately, the pairwise AUC-type risk can not satisfy this assumption. For instance, the optimization functions for the detector, $\ell_{sq}(r(x_b^i;\theta_r) - r(x_n^j;\theta_r))$ and $\ell_{sq}(r(\tilde{x}_b^{i};\theta_r) - r(\tilde{x}_n^j;\theta_r))$, are interdependent if any term is shared (e.g., $\tilde{x}_n^j = x_n^j$ or $\tilde{x}_b^i = x_b^i$). To this end, in this study, we use covering numbers and $\epsilon$-net arguments [3] in the subsequent proof to derive the generalization bound.

Q2: The authors should elaborate further on the calculation of the base-domain confidence score $r$.

Following the setting in [4], the new domain class names are known during testing in the OPT task and the base-domain confidence score is derived from the maximum probability over the base domain. A high value of $r$ means the high probability of the sample belonging to the base domain. Given the image_features [1,512] and text_features [512,C] where 512 means the dimension of the latent feature and $C$ is the number of all classes, we calculate the prob = Softmax(image_features·text_features,dim=1) and then base domain confidence score can be obtained via $\max_{1\le j\le C_b}**prob**[:,j]$, where $C_b$ is the number of base classes.

Q3: The ablation studies of the gating mechanism.

Following your suggestion, we further explore the effectiveness of the gating mechanism. Replacing the sigmoid-weighted gate with a fixed 0-1 mask ("Ours 0-1 Gate") slightly improves over removing the gate entirely ("Ours w/o Gate") but under-performs compared to the adaptive sigmoid gate. It validates the effectiveness of sparse sample selection mechanism and the gate approximation mechanism.

[1]Rademacher and Gaussian Complexities: Risk bounds and structural results. COLT, vol. 2111, pp. 224–240, 2001.

[2]Foundations of Machine Learning. MIT Press, 2012.

[3]Probability in Banach Spaces: Isoperimetry and processes. 1991.

[4]DECOOP: Robust Prompt Tuning with Out-of-Distribution Detection. ICML 2024.

Thanks for your valuable comments! Due to space constraints, we include full tables and figures https://anonymous.4open.science/r/R-4D06. References to Tab.X, Fig.X correspond to those provided in this link.

Q1: OpenworldAUC for fine-grained model capability analysis

The OpenworldAUC comprehensively evaluates: 1) base-to-new detection 2) base classification 3) new classification. A high OpenworldAUC indicates the model performs well on three. To diagnose low OpenworldAUC, we can further check three sub-metrics: AUROC, BaseAcc, NewAcc. To validate this, we present fine-grained results on four datasets shown in the table below and Tab.1, Fig.1, which reveal:

• OOD-focused methods (Gallop) excel at BaseAcc-AUROC but struggle with new domain classification
• Base-to-new methods (DePT) show weaker detection performance
• Our method achieves better OpenworldAUC, indicating improved trade-offs across three, which further validates the comprehensiveness of OpenworldAUC. |F102|Baseacc|Newacc|AUROC|OpenworldAUC| |-|-|-|-|-| |DePT|97.68|75.38|92.83|69.46| |Gallop|98.60|70.94|94.50|65.69| |DeCoOp|93.90|76.24|96.84|70.28| |Ours|96.53|77.16|96.95|72.79|

Q2: How to ensure the quality of prompts

The prompts of base classification and base-to-new detection can be automatically optimized via our carefully designed loss function, which is a paradigm widely validated in existing prompt tuning research. Additionally, the prompt of new classification is hand-crafted to alleviate overfitting on the base training set. Unlike the naive prompt template "a photo of {}", in our initial submission, we choose a more informative prompt template based on the base domain, further ensuring the new-domain classification performance, shown in the table below and Tab.2.

Q3: Computational complexity of GMoP

In fact, we have included the prompt complexity analysis in Fig.5 in the initial version. To highlight the efficiency of our method, we present those numerical results in the table below and Tab.3, along with additional results on inference speed.

• Our method outperforms SOTA methods on the average performance of three open-world tasks with a smaller parameter cost. While recent SOTA methods design deep prompt structures, we optimize multiple shallow prompts in detector and classifiers.

• We measure inference speed by comparing average processing times per sample across ten datasets, testing representative methods and ours. While slightly slower than CoOp due to prompt mixing, our approach runs 34% faster than DeCoOp and matches DePT/Gallop in speed(S) |CoOp|DePT|Gallop|DeCoOp|Ours| |-|-|-|-|-| |0.00117|0.00167|0.00148|0.00273|0.00180|

Q4: The choice of multiple loss function weights across different datasets

It's a pity that our method is not fully understood. In fact, our optimization framework involves only one loss weight λ of the CE regularization added to the AUC loss, which has been discussed in App.E.6 in the initial version. Sensitivity tests across four datasets show consistent performance when λ∈[1/2,1], with SOTA results in this range, shown in the table below and Tab.4

Q5: The generalization of pseudo base-new partition

The foundation model itself has strong base-to-new generalization ability. Our task is to enhance such ability by leveraging base training data. To this end, we adopt the following partition strategy to simulate new class detection.

• We perform multiple (K) base-to-new partitions to ensure statistical stability of the new class simulation

• We ensure K pseudo base classes fully cover the base class

• Thm.5.2 suggests increasing K reduces such approximation error. Experiments on two datasets confirm this shown in the table below and Tab.5. We usually set K=3 to balance performance and efficiency |SUN397|AUROC|OpenworldAUC| |-|-|-| |K=1|84.45|53.16| |K=2|87.57|56.14| |K=3|90.70|58.54| |K=4|91.02|58.63| |K=5|91.25|58.71|

Q6: Imbalanced sampling of base/new data may affect detection optimization

Test Imbalance: Since true new-class data is unavailable during training, test imbalance doesn't impact optimization

Pseudo Imbalance: We sample pseudo base/new pairs from true base for AUC loss training. AUC's distribution insensitivity ensures robustness to pseudo base-to-new imbalance, validated by stable performance with varying ratios on DTD

Thanks for your constructive comments, and we would like to make the following response.

Q1: The necessity of introducing the Gated Mixture-of-Prompts remains unclear. A more detailed explanation is needed to justify this design choice.

The three components $g$, $h$, and $r$ have conflicting objectives: $g$ classifies base samples (its objective function relies on base samples), $h$ classifies new samples (its objective function uses new samples), and $r$ ranks base-new pairs (its objective function uses base-new sample pairs). Using a single prompt leads to mutual interference. To address this, our design assigns distinct prompts to each component, separating their optimization processes. A gate mechanism, which uses sigmoid-weighted confidence scores, adaptively combines their outputs. This ensures that the $r$-prompt focuses on ranking correctly classified pairs. By doing so, we prevent conflicts and allow each prompt to specialize for its task. Empirical results from six datasets confirm the difficulty of optimizing OpenworldAUC with a single prompt and demonstrate the effectiveness of our mixture-of-prompts strategy.

Q2: How is ( OP_0 ) derived from Proposition 5.1? Additionally, why is it valid to approximate the 0-1 loss using the square loss? I believe the authors should provide a clearer explanation, along with relevant references, to aid readers who may be less familiar with this topic.

($OP_0$) is derived by replacing the population risk in Proposition 5.1 with its empirical approximation (since $\mathcal{D}$ is unknown). Besides, following the framework of surrogate loss, we replace the non-differential 0-1 loss with a convex loss function $\ell$, such that $\ell(t)$ is an upper bound of $\ell_{0,1}(t)$. Note that if the scores live in $[0,1]$, standard loss functions such as $\ell_{sq}(t) = (1-t)^2$ often satisfy this constraint. This smooth approximation enables gradient-based optimization while preserving ranking semantics. The details can also be found in the recent survey [1,2,3].

Q3: The motivation behind the zero-shot new domain classifier is not well articulated. A more thorough discussion would help clarify its role in the overall framework.

Thank you for your constructive suggestion! The motivation behind the zero-shot new domain classifier falls into the following two aspects.

Balance efficiency and accuracy: Recent prompt-tuning methods show that learnable prompts, optimized on the base domain with only CE loss, may hurt new-domain classification performance compared to zero-shot CLIP with fixed prompts. The results on the NewAcc of the Zeroshot CLIP and the PT baseline CoOp confirm this point, as shown in the table below.

To improve generalization, many recent methods focus on maintaining alignment (loss design) with zero-shot CLIP while using structured prompts (structure design) to preserve zero-shot knowledge. These methods slightly outperform zero-shot CLIP in term of new domain accuracy but introduce additional computational and storage overhead. To purse a tradeoff between the, we just design a hand-crafted prompt.

Decoupling Classification in different domains: Since the base-to-new detector effectively distinguishes base and new samples in open-world scenarios, we can decouple the learnable prompt $\theta_g$ (for base-domain classification) and the fixed prompt $\theta_{h}^*$(for new-domain classification) and achieve promising performance. In other words, using different prompt parameters for classification task in different domains is feasible and effective in practical scenarios.

To further validate the effectiveness of zero-shot new domain classifier, we replace $\theta_{h}^*$ with $\theta_g$ and observe the overall OpenworldAUC performance drops, confirming the necessity of this decoupling and the effectiveness of the zero-shot new domain classifier.

[1] Auc maximization in the era of big data and ai: A survey. ACMComputing Surveys (CSUR), 2022.

[2] On the consistency of AUC pairwise optimization. IJCAI, 2015.

[3] Stochastic auc maximization with deep neural networks. ICLR,2019.