A Generalization Theory for Zero-Shot Prediction

Thank you for your hard work in verifying the paper. The authors are happy to hear that you found the narrative insightful and well-written. We address your concerns below.

“[The Methods And Evaluation Criteria] make sense, even though they can be extended considerably… [The Experimental Designs Or Analyses] make sense but…are quite limited to classification… Heavy focus on theory and limited experimental results and demonstrations.”

We acknowledge this feedback by 1) motivating why this is the most relevant task for studying modern ZSP, 2) providing additional experimentation with simulations using CLIP/VICReg and linear probe baselines, and 3) by further justifying the theoretical focus of the work. Details are given below.

Firstly, we would like to emphasize that this task is by far the canonical one for studying ZSP. This is largely due to the fact that prompting is inherently tied to natural language. Indeed, consider Gadre et. al. (NeurIPS, 2023), one of the largest-scale studies on the design of multimodal pre-training data for foundation models. In over three hundred experiments, models are evaluated on either zero-shot image classification, zero-shot image/text retrieval, or linear probing. As in our case, only the encoder architectures, datasets, and prompting strategies are varied. In retrieval, there is no prompting, and in linear probing, training data is available.

Secondly, in response to the explicit requests of 8U7M, we also include linear probing comparisons in Figure 5 of the rebuttal. This gives an idea for the near optimal performance of the ZSP methods in the case there there was no residual dependence and that the prompting strategy was unbiased. How this performance gap depends on the residual dependence $I(X;Y|Z)$ is studied experimentally in Figure 6 of the rebuttal as well.

Finally, while you correctly pointed out that there is a relatively heavy focus on theoretical work in the paper, we highlight that this is an intention, as the mathematical foundations of modern ZSP is still in its infancy. During the time of submission, the only directly related work we were aware of was Chen et. al. (ICLR, 2024), which we comment on the bottom of Page 2. We were also made aware of the very recent preprint Oko et al. (arXiv, Jan. 2025) by Reviewer fhW7. Thus, we feel as though the theoretical focus of the work is apt based on the current gaps in the scientific literature on this topic.

Once again, we are grateful for your recommendation of acceptance and are open to address any additional comments or questions during the discussion period.

Thank you for your thorough review. We address your comments below.

“I hope the authors could discuss more on the relationship between FSL, ZSP and MAML.”

We discuss model agnostic meta-learning (MAML) in its offline variant, i.e., as a tool for multi-task or meta-learning. The offline meta-learning method can be thought of a simpler setting than FSL, in which the downstream evaluation tasks are given to the user upfront. Thus, pre-training an encoder and training a predictor for all of the evaluation tasks can be done in one end-to-end step. On the other hand, FSL is not (in general) given any downstream task upfront, so models cannot be learned in an end-to-end manner at once. ZSP is a "harder" setting still, as no downstream data is given at any point in the training-evaluation pipeline.

“Although Section 2 has demonstrate why ZSP is best compatible with multimodal contrastive learning, it does not necessary discuss the advantage of ZSP over other SSL methods. One reason could be that obtaining labels for FSL is hard, but I would like to see more evidence supporting this.”

First, we clarify that we do not intend to “discuss the advantage of ZSP over other SSL methods”, because SSL is precursor to ZSP; they are not comparable. However, for the comparison of ZSP to FSL, obtaining labeled data is precisely the bottleneck that motivates ZSP (as you correctly pointed out). Second, the authors request additional clarification on what “more evidence supporting this” refers to, so we may adequately address your points. Having no access to downstream training data is not a claim; it is a specific data availability regime that is now well-established a modern machine learning setting (see Pourpanah (2022)). ZSP is the corresponding pipeline for this problem. Thus, we do not argue for any advantage of ZSP over FSL—they are simply different methods for different problems. Accordingly, different SSL methods accompany these problems, as alluded to in Section 2.

In fact, we generally always expect FSL to perform better than ZSP with equal pre-training data, as FSL receives strictly more task information than ZSP. That being said, a question we do consider is 1) “how close can ZSP get to the performance of FSL (with a large amount of training data)?” and 2) “how do we quantify it?”. This exactly leads to our mathematical notions of prompt complexity and residual dependence. That is, if the prompting strategy approximates the conditional distribution of the text $Z$ given $Y = y$ and we have that $X$ and $Y$ are approximately conditionally independent given $Z$ (see Figure 2 in the paper), then ZSP has no theoretical disadvantage over FSL. In response to your review, we verify this in simulation in Figure 6 of the rebuttal.

“[Q1 + Q2] ...how are prompts generated in the baseline method... Why does the performance of the baseline stay exactly the same as more prompts are provided? Are the baseline prompts provided all at once?”

The baseline prompts are not “generated”, as they are selected by humans as defaults in the CLIP benchmark package. Therefore, they cannot be created in arbitrarily large amounts such as LLM-generated prompts. Rather than matching the number of prompts between human baselines and LLMs, this illustration shows the scaling of the downstream classification accuracy as the user generates a large volume of prompts. Moreover, the goal of the experiment is not to market class-conditional prompting as a new method, but to experimentally verify the saturation point at which $O(1/M)$ prompt variance term in Theorem 1 becomes negligible.

[Q3] “The experiment does not compare ZSP with FSL baselines. I hope the authors could provide a sound reason for not doing so. To me, demonstrating the success of ZSP requires analysis and/or experiments to show it defeats other SSL methods… Why not compare CuPL/ZSP methods to these baselines?... the paper needs to demonstrate the advantages of CuPL/ZSP over other SSL methods, but not just community-curated prompts.”

As mentioned above, demonstrating the success of ZSP as compared to FSL was not a goal of this paper. Similarly, SSL was not a baseline for ZSP to defeat, but rather a form of pre-training that may lead to the ZSP capability. However, we are interested in studying the performance gap and its dependence (see Figure 6). We provide experiments that compare the ZSP performance to FSL baselines using linear probing on the evaluation datasets seen in the paper: FGVC Aircraft, DTD, Flowers 203, SUN397 (see Figure 5).

If we have addressed your concerns, we would appreciate that your score be raised, and if not, we are happy to answer any questions or consider additional experiments!

Thank you for taking the time to read our manuscript critically. This is undoubtedly one of the most comprehensive reviews we have ever received. Please see your comments addressed below.

“What are the implications of prompt bias and residual dependence?... Clarifying their role... would strengthen the paper's contributions… would it be possible to construct a synthetic dataset that aligns with the theoretical assumptions?”

We provide a synthetic example (see Figure 6) to highlight these implications. We focus on residual dependence, which has only been previously alluded to in Oko et al (arXiv, 2025), but simply assumed be zero. We aim to clearly illustrate two claims: 1) the residual dependence $I(X;Y|Z)$ (for image-caption-label $(X, Z, Y)$) governs the performance gap between the two-stage predictor (Eq. (5) / Eq. (6) of our paper) and the Bayes optimal predictor and 2) the encoder-based zero-shot predictors used in practice (such as in CLIP) behave as the two-stage predictor given enough data. The mathematical details are given in the linked derivations. This experiment controllably interpolates between the “worst” setting $I(X;Z|Y) = 0$, where ZSP performs at near chance and $I(X;Y|Z) = 0$, where ZSP performs optimally. While further investigation would be interesting, both claims are supported within the simulation.

In other words, $I(X;Y|Z)$ is a simple distribution parameter that governs how close the best ZSP method can do compared to the optimal downstream predictor, which is of clear interest both in theory and practice. While out of scope for this paper, we hypothesize that this quantity can be estimated in pre-training data selection methods.

“… are the authors proposing a new loss function for ZSP…? … it remains unclear whether this [variance-regularized covariance] formulation offers practical benefits over existing self-supervised learning objectives.”

There is no intention in the paper to propose a new loss function for SSL, or to even promote one SSL method over another. On the contrary, we include Appendix E to embrace existing, complementary work on SSL and to provide intuition on the relationship between our normalized cross-covariance-based estimator and the common SSL procedures used in practice. This is verified experimentally at least for the motivating example of CLIP in the given simulation.

“zero-shot image classification benchmarks… are reasonable choices for proof-of-concept experiments on prompt complexity… the analysis does not extend to other self-supervised learning (SSL) settings.”

Due to character limits, please see our response to Reviewer GT2x, in which we describe the motivation for this task. In summary, because we specifically study zero-shot prediction through prompting, similar tasks such as image/text retrieval (which do not include a prompting component) are not meaningful in our setting.

“…what exactly does "ideal prompting" refer to?.”

The ideal prompting strategy is one in which the prompt bias term is zero, i.e. the user is able to draw from the distribution $P\_{Z|Y=y}$ (one of the implications of defining prompt bias in the first place). In the ImageNet-Captions dataset, we are able to compare to this ideal strategy because we observe direct observations of $(Y,Z)$ pairs (as images have both captions and labels). Thus, we hold out a pool of pre-training examples and draw from these pairs to estimate the prompts.

“Moreover, the paper explicitly poses the question: "...does ZSP achieve near-optimal statistical performance on a downstream task, and with what dependence on (1) the pre-training data distribution, (2) the downstream task distribution, and (3) the prompting strategy?" However, the empirical experiments are not structured to directly address this question.”

Thank you for raising this point. On (1), our theoretical analysis suggests that this dependence is captured by the residual dependence quantity $I(X;Y|Z)$, and upon your suggestion, we designed synthetic experiments to address this claim. We can perform similar real data experiments if the reviewer finds it helpful. On (2), we did not dedicate experiments to showing how distribution shift may affect performance, as countless empirical studies (see Quiñonero-Candela et al. (2022)) exist on this topic. On (3), the experiments in Section 4 help determine the scaling behavior of the accuracy with the number of prompts, highlighting that the ideal range for reducing prompt variance can be as high as 50-100.

We also have addressed all other comments (references, clarifications) but reserve them for the discussion period due to space limitations. Given our efforts to improve the paper in experimentation and presentation in light of your recommendations, we hope you will consider raising your score to above the acceptance threshold. Please allow us to answer additional concerns you have!

Thank you for your review—we address your main comment below.

"The theoretical analysis is limited to CLIP-like multi-modal models."

Our analysis broadly describes multimodal encoder + prompting strategies, where the encoders could be learned by a variety of objectives (not only CLIP). Please see Appendix E for a review of such multimodal encoder strategies, including Multimodal InfoNCE/CLIP, BarlowTwins/Nonlinear CCA, the Spectral Contrastive Loss, and Multimodal VICReg. Crucially, we do not focus on the mechanics of how particular objectives or optimization algorithms result in particular encoders (see the related work Section 1 and references therein). We focus on the downstream performance is affected by 1) the dependence structure of $(X, Z, Y)$, the image-caption-label triple, and 2) the nature and amount of the prompting strategy.

To improve based on your feedback, we have included a simulation in Figure 6 of the rebuttal wherein the two-stage predictor analyzed in our paper is compared to both CLIP and VICReg trained on the corresponding data. We find that the dependence of two-stage prediction on the residual dependence follows the same trend as that of both CLIP and VICReg.

If this point is addressed adequately, then we would appreciate that the score be raised; if not, please let us know if you have further comments or questions and thank you once again for this feedback!