Boosting Vision-Language Models with Transduction

W1: Unclear motivation.

While VLMs such as CLIP enable zero-shot predictions, they are far from being perfect (please refer to the zero-shot performances of CLIP for 6 different model sizes in Table 8 in the Appendix). Please note that CLIP receives unlabeled samples and makes class predictions by comparing the images to several candidate text prompts. Therefore, CLIP yields pseudo-labels that could be used as a noisy supervision in transduction without any label.

What we propose in our work is to improve zero-shot CLIP and inductive few-shot methods based on CLIP as pre-training, by incorporating the transduction paradigm during inference.

W2: Mismatch with Zero-Shot Learning.

It is true that many transductive, vision-only methods (e.g., [5, 13, 33, 36, 43, 73, 24 , 72], among others) propose to propagate knowledge from observed training (labeled) data to unlabeled test data. However, in this work, we show that, in the context of VLMs, transduction could also be conducted without any label, by leveraging the pseudo-labels generated from the text embeddings, as a form of noisy supervision.

Indeed, we make zero-shot predictions by employing text-based noisy supervision (the KL term in Eq. 2) and the structure of the unlabeled data in the feature space (GMM clustering and Laplacian regularization terms), which effectively corresponds to the transductive paradigm. Please note that we propose an extension of TransCLIP, named TransCLIP-FS (the objective function in Eq. 4), for cases where labeled data are available, which combines noisy supervision from the zero-shot predictions and supervision from the few-shot samples, as in the above-mentioned transductive methods.

W3: Lack of Intuitive Justification.

Thank you for giving us the opportunity to clarify. The GMM term could be viewed as a maximum-likelihood estimation objective, like in the Expectation-Maximization algorithm. It could also be viewed as a probabilistic generalization of the K-means clustering objective (lines 157-162). The principal difference with the standard K-means clustering lies in the capability of TransCLIP to learn a covariance matrix (variable $\boldsymbol{\Sigma}$).

As shown in Figures 2a and 2b, which we added for this rebuttal (please refer to the attached PDF document), the learned covariance matrix enables fitting the unevenly spread feature distribution across the dimensions of the embedding space. We also show that fixing the covariance matrix (which corresponds to K-means if we omit the Laplacian and KL terms) results in a performance drop (line 2 in Table 6a of the main paper).

W4: Computational Overhead.

The Block Majorize-Minimize procedure presented and detailed in Section 3.3 provides a decoupled $\mathbf{z}$-update equation (Eq. 5), as well as closed-form updates for GMM parameters $\boldsymbol{\mu}$ and $\boldsymbol{\Sigma}$ (Eqs. 6 and 7). These decoupled updates enable the algorithm to run substantially faster than popular prompt-tuning approaches (such as UPL [25]), which require costly forward-backward propagation for multiple epochs. We report run-time comparisons in Tables 5a and 5b in the main paper.

Furthermore, we add Table 1 in this rebuttal (please refer to the attached PDF document), which provides a more detailed analysis, showing: (i) TransCLIP's prediction time is a fraction of the time required for feature encoding, which makes its total runtime (encoding + prediction) comparable to the total runtime of the inductive zero-shot CLIP baseline; (ii) The total runtime of UPL is an order-of-magnitude slower, due to the training overhead. We will add these detailed runtimes to Tables 5a and 5b in the paper.

W1: Balance of terms.

We understand your concern. The terms in our objective function are not necessarily in conflict; they could be seen as a clustering (driven by the GMM term), which is regularized by the Laplacian term propagating the labels between neighboring samples, and by the KL-divergence term that incorporates the text knowledge of the VLM.

Regarding the utility of the two regularization terms, we show in Table 6a (line 3) of the main paper that the Laplacian term helps improving performance. In Table 4 of the main paper, we demonstrate the importance of the KL text regularization (TransCLIP vs. TransCLIP w/o text).

Regarding the sensitivity to hyper-parameters, we emphasize that the text-regularization weighting factor $\lambda$ is set to the same value for all the experiments that involve TransCLIP-ZS, regardless of the dataset or the few-shot method. We evaluate a posteriori the impact of varying $\lambda$ and show that there is only a minor sensitivity as long as $\lambda$ does not tend towards $0$ (please refer to Table 6b of the main paper, and to Figure 1 that we added in this rebuttal, in the attached PDF document). We also evaluate the sensitivity to the number of neighbors considered in the Laplacian term, and show again that it does not significantly impact the performances (please refer to Table 6c of the main paper).

W2: Computation costs.

• Time complexity: The Block Majorize-Minimize procedure detailed in Section 3.3 presents a decoupled $\mathbf{z}$ update equation (Eq. 5) and closed-form update equations for the GMM parameters $\boldsymbol{\mu}$ and $\boldsymbol{\Sigma}$ (Eqs. 6 and 7). These enable our algorithm to achieve rapid convergence compared to traditional prompt tuning methods such as UPL [25], which require extensive forward-backward propagation over multiple epochs. We provide runtime comparisons in Tables 5a and 5b of the main paper, and a more detailed analysis in Table 1 of the attached document. We will incorporate these new results into Tables 5a and 5b of the main paper to clarify the speed and efficiency advantages of TransCLIP.

• Resource usage: we discuss the hardware requirements of TransCLIP in Appendix B, noting that TransCLIP consumes 16.9 GB of memory when inferring on ImageNet, making it feasible to run on a single 24 GB GPU for large datasets.

W3: Scalability to even larger datasets.

As discussed in Section 4.2, TransCLIP is easily applicable to multi-billion parameter models and large datasets since it does not necessitate gradient computation or model parameter training (i.e., it only requires the memory needed for single-sample inference, because the whole dataset processing can be performed one sample at a time). Results in the Table below support the scalability of TransCLIP in effectively managing more than one million images in a few minutes (on ImageNet training set which contains 1,281,167 images categorized into 1,000 classes).

W4: Theoretical convergence guarantees.

As stated in Section 3.4, our optimizer can be viewed as an instance of the general Block Majorize-Minimize (BMM) paradigm, and we establish convergence of our procedure. The technical conditions that guarantee convergence stem from the mathematical properties of our objective and its block-wise majorizing functions used in the BMM optimizer (such as the strong convexity of the block-wise majorizing functions). Hence, these conditions do not restrict any practical application aspects.

Q1.a: Online setting.

This is indeed a general limitation of transductive-inference approaches. Still, we believe this transductive setting is useful in a breadth of application domains and real scenarios, as pointed out by Reviewers NaRY and uSPt and evidenced by an abundant and recent literature on transductive few-shot learning; actually, in the recent years, the transductive setting has been very popular in the vision-only few-shot learning literature (see, for instance, [5, 13, 33, 36, 43, 73, 24, 72], among others). Use cases where batches of test imaging data are available occur frequently in practice, e.g., in video-stream sequences, or in smart-device photos taken and stored every day. The setting is also very appealing in various application domains, such as remote sensing or histopathology where large images are divided into patches, or more generally, to analyze existing databases or large corpora of web-scraped unlabeled images (e.g., to create pseudo-labels for large unlabeled image sets taken and stored every day).

As correctly mentioned by the reviewer, if data points are coming sequentially, we would need to wait to collect a batch of them before conducting TransCLIP (while still using the zero-shot ability of the VLM if on-the-fly prediction is needed). We will add this discussion to the limitation section, as well as to the conclusion, as this could be an interesting future work to broaden the application range of TransCLIP.

Q1.b: Prediction for new samples.

This is, indeed, a very interesting point and extension that we missed in our first version. In fact, after running TransCLIP, the GMM parameters could be stored to make a prediction on a new individual sample, resembling the inductive paradigm:

$$\mathbf{z}_{i} = \frac{ \mathbf{p} _{i} }{ \mathbf{p} _{i}^\top \mathbf{1}_K }$$

This can be seen as a Maximum Likelihood predictor for sample $i$. We denote this naive prediction rule $***MLH***$.

More interestingly, Update Eq. 5 can be directly adapted. Let's assume we don't have access to other data points (e.g., very low resource application or for privacy concerns), we remove the Laplacian term and get the following prediction rule:

$$\mathbf{z}_{i} = \frac{\hat{\mathbf{y}} _{i} ^ {\lambda} \odot \exp ( \log ( \mathbf{p} _{i}))}{(\hat{\mathbf{y}} _{i} ^ {\lambda} \odot \exp (\log (\mathbf{p} _{i}))) ^ \top \mathbf{1}_K} = \frac{\hat{\mathbf{y}} _{i} ^ {\lambda} \odot \mathbf{p} _{i}}{(\hat{\mathbf{y}} _{i} ^ {\lambda} \odot \mathbf{p} _{i}) ^ \top \mathbf{1}_K}$$

This can be seen as a Maximum A Posteriori predictor for sample $i$ since it is the class probability likelihood $\mathbf{p}_i$ weighted by the initial pseudo-label $\hat{\mathbf{y}} _{i}^{\lambda}$ (prior knowledge). We denote this prediction rule $***MAP***$.

Following this very interesting comment by the reviewer, we conduct new experiments (average over 100 random seeds): We split the test set, keeping randomly 10% of the data points as a held-out set, and infer TransCLIP on the remaining 90% of the samples. We then apply the prediction rule on each sample of the held-out set (independent predictions, as in the inductive setting). The Table below summarizes the performance and shows that our $***MAP***$ predictor rule for new unseen data points still brings a significant gain, validating the feasibility of this approach.

Q2: Precision on runtime.

We provide more details on the runtime of each step in the attached document (see Table 1). On ImageNet, UPL$^*$ takes 151 minutes for prompt tuning, followed by 59 seconds for images+texts encoding during the inference step, resulting in a total of 152 minutes. In contrast, TransCLIP has no training stage and requires 14 seconds to run, which is only a fraction of the 59 seconds needed for images+texts encoding. We will extend Table 5 of the main paper to include these clarifications.

Q3: Comparison to unsupervised learning.

We refer to Table 8 in the Appendix, which contains a direct comparison of TransCLIP with UPL [25]. Please note that UPL was initially designed to find pseudo-labels in a unlabeled training set, followed by a training stage before inferring on the test set, making UPL an inductive method. Therefore, we extended UPL to UPL$^*$, drawing pseudo-labels directly from the test set, hence making it a transductive method (please refer to the details in lines 569-572). In conjunction with our response to W6 by Reviewer 1 (NaRY), we will add a description of UPL and UPL$^*$ in the Appendix for better clarity.

We also compare TransCLIP to other unsupervised methods (in Table 8 of the Appendix) such as TPT [42] and MTA [65] (two test-time augmentation techniques) and SwapPrompt [40]. We draw attention to the differences between their settings and ours (lines 567-569), which will be extended by discussion "W1 and Q2" with Reviewer aJGT.

W1 and Q2: Discussion on Test-Time methods.

We agree that the mentioned test-time methods also employ the transduction paradigm. However, their settings are in fact very different from the ones studied in our work (zero- and few-shot adaptation of CLIP, improving inductive few-shot methods). We still report their performance in Table 8 of the Appendix but we agree that a more complete discussion is needed. We will add the following discussions:

• Design: SwapPrompt [40] has been designed to make batch predictions on-the-fly, and has continual-learning related mechanisms such as an exponential moving average prompt across batches. The setting of TPT [42] also differs from ours; it has been designed to work on a single sample with many data augmentations to train one prompt per image.

• Resource: Both methods require access to model weights for training (i.e., do not operate in a black-box setting). We also note that prompt tuning does not scale well with the model size and is even impractical on very large models such as EVA-CLIP-8B.

• Computation overhead: TPT takes nearly 5 hours for ImageNet. As SwapPrompt relies on prompt learning for several epochs and requires data augmentation, its running time is above the one of UPL$^*$ (please see Table 1 in the attached document). In comparison, TransCLIP takes only 73 seconds for the whole pipeline (images and texts encoding + transduction; please refer to Table 1 in the attached document).

• Batch size: It is true that TransCLIP first requires a test batch to work on. However, we would like to highlight our answer Q1.b for Reviewer AvCq, which discusses and assesses how online predictions for new (single) samples can be processed after running TransCLIP.

W2: Issues with missed references. All Section 3.3 follows similar arguments from [1], but re-derived by adding the language term.

Please note that we mention [1] (LaplacianShot, reference [73] in the paper) in related work (lines 85-87), and provide direct experimental comparisons to it in Table 4 in the paper, as well as in Tables 17 to 21 in the Appendix. We agree with the reviewer that we could emphasize that the derivation of [73] (i.e., linearization of the Laplacian term) is a sub-step of our approach but without the text term (this could be done in sub-section Majorize-Minimize with respect to the z-block in section 3.3).

We clarify, however, that [73] does not have a Block Majorize-Minimize (BMM) optimization procedure (i.e., the additional $\boldsymbol{\mu}$ (prototypes) and $\boldsymbol{\Sigma}$ (covariance) steps in Eqs. (6) and (7) in our section 3.3). The objective function in [73] is quadratic (the term using the fixed prototypes being linear) and has only assignment variables, whereas our objective function is higher-order with three blocks of intertwined variables (prototypes, covariance and assignments). Interestingly, Table 4 shows that, even without the text term, our objective function brings significant improvements over the LaplacianShot one when the number of shots increases (due to learning prototypes $\boldsymbol{\mu}$ and covariance $\boldsymbol{\Sigma}$ from both unlabeled and labeled data).

We would also like note that the general principle of linearizing concave quadratic terms (which provides a majorizing function on the Laplacian term in our case) is quite common in the Majorize-Minimize optimization literature, and has been used in other works in machine learning, even much earlier than LaplacianShot, e.g., the following work in the context of conditional random fields (CRFs): Krahenbuhl and Koltun, Parameter Learning and Convergent Inference for Dense Random Fields, ICML 2013. We will add this reference and connect to the LaplacianShot procedure ([73]) in Section 3.3 (e.g., at line 193).

W3 and Q1: GMM argument.

They are two major differences with the linear term $\mathcal{N}(Y)$ in LaplacianShot [73], in which the class prototypes are fixed (computed from the labeled shots only) and the assignment vector is the only block of variables. First, as mentioned by the reviewer, our formulation enables to learn class prototypes $\boldsymbol{\mu}$ from both the unlabeled and labeled data (both in zero- and few-shot). This claim is supported by the results in lines 1 and 4 of Table 6a in the main paper, which show that performance tends to decrease when the class prototypes are fixed.

Second, by introducing a learnable covariance matrix $\boldsymbol{\Sigma}$, we enable TransCLIP to better model the feature variance, which may be unevenly spread across the dimensions of the embedding space. To further support this claim, we add Figures 2a and 2b in the attached PDF document, which show that feature variance varies across dimensions, and that our learnable $\boldsymbol{\Sigma}$ tends to fit this particular distribution. You are correct that learning a covariance matrix is equivalent to learning the weights of the Mahalanobis distance between the data points and their class prototypes, and we will add this discussion to Sec. 3.1, in the GMM-based clustering part. Thank you for pointing that out! Interestingly, the learned GMM parameters can then be used to make predictions on new incoming samples, as discussed in Q1.b for Reviewer AvCq.

Q3: KL Div parametrization.

We would like to point to Table 6b in the main paper, which studies the impact of $\lambda$ for the zero-shot setting.

However, this Table studies only the zero-shot case. Therefore, we provide here a more thorough study (please refer to Figure 1 in the attached document) for a wider range of settings (zero-, 1-, 4-, and 16-shot), which clearly shows both the importance of this term and its stability around the selected value. Indeed, the average zero-shot performance does not vary significantly between $\lambda=0.5$ and $\lambda=1.6$, and $\lambda=0.5$ is an acceptable value for all few-shot settings. We will add this extension to our current ablation studies.

W1: Eq 2.

It's an omission on our side; indeed, it should be $\mathbf{z}_i^\top \mathbf{z}_j$. Thanks for pointing that out! We will update the objective function accordingly.

W2: Final prediction.

Yes, we take the argmax for the final prediction. We should have made it clearer at the end of Section 3.3 as follows: `'Note that, after convergence, we use the sample-to-class assignment variables $\mathbf{z}_i$ as predictions for each sample $i$ of the query set $\cal Q$, using the argmax operation for conventional classification.". We will also update Algorithm 1 in the Appendix to further clarify this (line 13: return argmax($\mathbf{z}$)).

W3: Design choice (using a GMM).

We agree that further discussions / illustrations on this choice would be helpful to readers. The most basic unsupervised clustering objective would be K-means. This could be viewed as an extension of inductive methods (e.g., CoOp, which also search for new class centroids via prompt tuning) to unlabeled test data. However, K-means makes the assumption that the clusters are 'spherical', which may not be the case for the embedding vectors ensuing from VLMs. Indeed, K-means corresponds to a particular (simplified) case of our formulation, in which the covariance matrix is fixed to the identity matrix, as pointed out in lines 160-161.

The ablation studies in Table 6a show TransCLIP performance when $\boldsymbol{\Sigma}$ is not updated (note that the other regularization terms are present), resulting in a large performance drop. This suggests that the spherical-cluster assumption (i.e., identity covariance matrix), as in K-means, may not be valid for VLM embeddings. Table 6d also shows the impact of having an isotropic $\boldsymbol{\Sigma}$, which could be seen as an intermediate solution between K-means (with fixed $\boldsymbol{\Sigma}$) and the TransCLIP objective function (with learnable $\boldsymbol{\Sigma}$), demonstrating the importance of learning $\boldsymbol{\Sigma}$.

To provide further insights, we added Figures 2a and 2b to this rebuttal (please refer to the attached PDF document), which show that feature variance varies across the dimensions and that our learnable $\boldsymbol{\Sigma}$ tends to fit such 'elliptical' cluster distributions. Additionally, the introduction of a learnable $\boldsymbol{\Sigma}$ can also be seen as learning the weighting factors of the Mahalanobis distance between the class centroids and the data points (as pointed out by Reviewer aJGT).

W4: Affinity matrix.

You are right; our affinity matrix W can be rewritten as $F^\top F$ and is indeed a Gram matrix, hence PSD. Therefore, the max operator is not necessary to ensure the PSD condition. We will remove this statement. Thanks!

W5: Effect of Laplacian regularization beyond the simplex vertices.

Yes, the effect of Laplacian regularization holds within the simplex, beyond the vertices. Indeed, for a given, non-vertex $\mathbf{z}_i$ lying within the simplex, the vector $\mathbf{z}_j$ that maximizes $\mathbf{z}_i^\top \mathbf{z}_j$ is the one-hot (vertex) vector where the entry corresponding to the argmax of $\mathbf{z}_i$ is 1 (e.g, for $\mathbf{z}_i = [0.7, 0.2, 0.1]$, $\mathbf{z}_j = [1, 0, 0]$ maximizes the dot product). This function is convex in $\mathbf{z}_j$ over the simplex, and any deviation of $\mathbf{z}_j$ from the maximizing vertex can only cause a decrease in the dot product. Of course, this is an intuitive example, but the general behavior of the Laplacian term in our objective function is more complex. Multiple points are linked to each other, with potentially different predictions (and different most confident classes), combined with the KL term penalizing divergence from the zero-shot prediction (potentially deviating from the one-hot solutions).

W6: Modification of UPL.

As suggested, we will add a small description of UPL and UPL$^*$ in the Appendix (in C.1 Zero-shot, line 570) for better clarity.

UPL: Unsupervised Prompt Learning [25] where N = 16 hard pseudo-labels per class are generated from a training set. Then a cross-entropy loss function on soft tokens is minimized (see UPL$^*$ below).

UPL$^*$: We implement a natural extension of UPL. N = 16 hard pseudo-labels per class are generated from the query (test) set ${\cal P}\subseteq {\cal Q}$ according to the prediction’s confidence. For fairness, we reevaluated the number of pseudo-labels to select and still found that 16 per class yields the best results on average, as seen in Table 22. The following cross-entropy loss function is minimized:

$$\mathcal{L}(\overline{\text{V}}| \\{ \mathbf{x}_i \\} _{i=1} ^{|{\cal P}|})) = \frac{1}{|{\cal P}|} {\sum} _{j \in {\cal P}} \mathcal{L} _{\tiny{UPL}} (\overline{\text{V}}|\mathbf{x}_j)$$

where $\overline{\text{V}}$ denotes the vector of learnable context token embeddings. At inference, the learned tokens are used to generate the text embedding of each class.