Whitened CLIP as a Likelihood Surrogate of Images and Captions

We thank the reviewer for their constructive feedback and comments. Below, we address key concerns regarding the applicability of our approach to zero-shot settings, and address the reviewers questions.

Methods and Evaluation Criteria

To address the reviewer’s comment regarding zero-shot transfer to a downstream task, we conducted a large-scale experiment on the generated image detection task. The generated images were synthesized from text, taken from three benchmarks [1,2,3]. Images are generated using 20 generative models, totaling 100k generated images. Since these datasets include fewer real images, we supplemented real samples from the MSCOCO training set.

Here, zero-shot refers to having no exposure to generated content and no task-specific training. We use only real images (from MSCOCO validation set) to compute the whitening matrix and do not fine-tune on this task. Thus, we compare against other zero-shot image detection baselines.

We benchmark against four zero-shot detection methods: AEROBLADE [4] (CVPR 2024), RIGID [5] (arXiv 2024), ZED [6] (ECCV 2024), and Manifold-Bias [7] (ICLR 2025). We use official implementations for [4] and [7], and self implement [5] and [6] based on their papers and available code (of other models used in these methods).

Each method outputs a continuous criterion score, which we binarize using a threshold from a small calibration set of 1k real images:
th = mean(C) + std(C), where C denotes criterion values. This calibration set is disjoint from the evaluation set.

We use four metrics: AUC and AP (Average Precision) as separation metrics, and F1-score and Accuracy as classification metrics. Evaluation is based on 100k generated and 100k real images. As shown below, our method outperforms all baselines across all metrics. This setting follows the protocol used in [7], the most comprehensive of the compared works. Our results for baseline methods align with those reported in [7]. We believe this validates our method’s ability to distinguish real from generated images in a zero-shot setting.

Performance Comparison

While on some generative models methods [6] or [7] occasionally perform competitively, ours is more consistent across all generative models, whereas others experience significant drops on specific ones. Full per-model results are available here:
https://drive.google.com/file/d/1hQFwAqpo3opByTqC70mb4-fDz0z8TkVQ/view?usp=drive_link

Theoretical Claims

When CLIP embedding features are highly correlated, the covariance matrix can become ill-conditioned, yielding near-zero eigenvalues. During whitening, eigenvectors are scaled by the inverse square root of these values, so eigenvalues close to zero can result with a numerically unstable and non-invertible whitening matrix W.

Questions for Authors

1. We thank the reviewer for highlighting two relevant papers, which we will cite in the final version. These works propose alternative, hierarchical embedding spaces to CLIP. The whitening transform is agnostic to the embedding space, as long as the data adequately represents it. In the context of these embedding spaces we assume this requires diverse samples, including many complex examples. For our likelihood approximation, a key factor is the distribution of each feature. As shown in Fig. 4.c, CLIP features exhibit a Gaussian-like distribution, which becomes standard normal after whitening, as validated in Tab. 1. If the original features of these embedders follow a different distribution, whitening may not yield standard normal features, undermining the assumptions of our likelihood model.

2. This question is addressed in the zero-shot detection experiment discussed above.

We hope these additional results and clarifications further support our proposed approach.

[1] Wang, Sheng-Yu, et al. "CNN-generated images are surprisingly easy to spot... for now." CVPR 2020.‏
[2] Ojha, Utkarsh, et al. "Towards universal fake image detectors that generalize across generative models." CVPR 2023.‏
[3] Zhu, Mingjian, et al. "Genimage: A million-scale benchmark for detecting AI-generated image." NeurIPS 2023
[4] Ricker, Jonas, et al. "AEROBLADE: Training-free detection of latent diffusion images using autoencoder reconstruction error." CVPR 2024
[5] He, Zhiyuan, et al. "RIGID: A training-free and model-agnostic framework for robust AI-generated image detection." arXiv 2024
[6] Cozzolino, Davide, et al. "Zero-shot detection of AI-generated images." ECCV 2024.‏
[7] Brokman, Jonathan, et al. "Manifold induced biases for zero-shot and few-shot detection of generated images." ICLR 2025

We value the reviewer’s critical insights and suggestions. Below, we provide detailed clarifications and supporting evidence addressing the concerns raised.

Experimental Designs or Analyses

Text Complexity

One example in Fig. 5 does show higher likelihood for a shorter sentence (left), but this does not imply a bias toward shorter inputs. In Fig. 7c, our method's likelihood is shown to be unaffected by caption length, unlike the other LLMs and VLMs. Thus, our findings are contrary to the reviewer’s claim. In fact, in Fig. 7a and Tab. 4, we present a surprising finding: LLMs and VLMs show similar distributions for captions with and without nouns. As illustrated, the noun-free captions are semantically illogical. Since LLMs and VLMs are highly biased toward shorter inputs, the distribution remains unchanged. In contrast, our method—agnostic to caption length—assigns significantly lower likelihood values to the noun-free captions.

Regarding cosine similarity as an alternative to likelihood: the two serve fundamentally different roles. Cosine similarity compares two inputs and outputs a scalar based on angular distance, whereas likelihood evaluates a single input. Therefore, cosine similarity is not suitable as a substitute for likelihood in this context.

Uniformity

• Unit Variance and Zero Correlation: As the reviewer notes, these properties are directly achieved through the whitening transform. While this is given in theory, this experiment empirically confirms that the transformation results in an isotropic space where features have zero mean, unit variance, and are uncorrelated. These characteristics, along with the normality verified in Sec. 3.3, are essential for computing likelihood using a multivariate Gaussian model (Sec. 3.4).

• Uniformity: We agree that we did not elaborate on uniformity in the main paper, due to space constraints. Here we supply a more detailed explanation - uniformity in latent representations ensures that embeddings are evenly distributed across the latent space, preventing representation collapse where different inputs become indistinguishable due to overly similar embeddings [1]. A uniform space enhances discrimination between inputs, improving downstream tasks such as classification [2]. It also promotes generalization and avoids overfitting to specific latent regions [3]. In contrastive learning, uniformity complements alignment by pushing dissimilar inputs apart [1], resulting in a well-structured, robust, and generalizable representation space. We will add a more detailed explanation to the appendix and refer to it from the main paper.

Data Analysis

We refer the reviewer to our new experiment on zero-shot generated image detection (see response to Reviewer 531Y). Specifically in relation to cosine similarity, we note that both RIGID and Manifold-Bias use cosine similarity as part of their detection pipelines. However, our likelihood-based method is simpler, significantly faster (see table below), and outperforms both baselines. Moreover, other cosine-based methods [4,5] for this task are not applicable in a zero-shot setting. This experiment highlights both the effectiveness and practical advantages of our approach. To demonstrate the efficiency of our method, we report per-image inference times on a single A100 GPU for the zero-shot detection task:

Other Strengths and Weaknesses

Not "Training-Free"

The reviewer raises a valid point: PCA in the whitening transform can be seen as unsupervised training. However, this is analogous to CLIP pretraining—performed once on unlabeled data and reused across downstream tasks. In our case, this step is extremely lightweight (under 1 second). As demonstrated above, leveraging the likelihood for a downstream task results with a very efficient method.

We hope these clarifications and new experiment address the reviewer’s concerns and underscore the practical strengths of our approach. We would appreciate positively considering increasing the final rating of the paper.

[1] Wang, Tongzhou, and Phillip Isola. "Understanding contrastive representation learning through alignment and uniformity on the hypersphere." ICML 2020
[2] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." ICML 2020
[3] Balestriero, Randall, and Yann LeCun. "Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods.” NeurIPS 2022
[4] Cozzolino, Davide, et al. "Raising the Bar of AI-generated Image Detection with CLIP." CVPR 2024
[5] Sha, Zeyang, et al. "De-fake: Detection and attribution of fake images generated by text-to-image generation models." ACM SIGSAC 2023

We appreciate the reviewer's detailed and thoughtful feedback. Below, we address each point raised.

Methods and Evaluation Criteria

First Point, first part

"While we have qualitative examples in Fig 2 and Fig 8 to show that W-CLIP can detect artifacts in synthetic images, we don't have a quantitative metric to know how good it is."

To address this, we conducted a large-scale experiment on zero-shot detection of generated images. Full details are provided in our response to Reviewer 531Y. A summary of the comparative results is given below:

First Point, second part

"Moreover, can W-CLIP embedding be used to tell the aesthetics or quality of images?"

We refer to Fig. 3 (top right), where an ImageNet-C experiment shows that norms of whitened embeddings respond to image corruptions (e.g., impulse noise). Additional results appear in Fig. 10 (Appendix B), covering 12 corruptions including blur, defocus, and low contrast. In all cases, corrupted images exhibit higher norms (in the whitened space) than the original images. As explained in Sec. 4.2 (lines 316–322), higher norms in the whitened space correspond to lower likelihood values, indicating reduced image quality. These results suggest that W-CLIP embeddings capture information related to image quality.

Second Point

"When using full circle SLERP to do image manipulation, we have qualitative examples in Fig 20 and Fig 21, but again, we don't have any large-scale evaluation to show the superiority of W-CLIP compared with CLIP."

To provide quantitative insight, we conducted a full-circle SLERP experiment on the MSCOCO validation set (5k images). For each image, we performed full-circle SLERP in both the CLIP and W-CLIP embedding spaces. In this process, a source image is interpolated toward a destination image along a circular path within the embedding space. Crucially, the image generated at the 180° position from the source—referred to as the “opposite image” (generated from the “opposite embedding”)—is invariant to the chosen destination and determined solely by the source. While other positions along the path are influenced by the destination embedding, the 180° embedding is a fixed, symmetric counterpart.

We generate these opposite images using both CLIP and W-CLIP embeddings and observe a stark contrast: in the CLIP space, opposite images degrade into structured noise, whereas in the W-CLIP space, they remain visually natural and semantically meaningful, as shown in Figs. 20, 21.

The structured noise produced by CLIP exhibits 4×4 pixel blocks and a restricted color palette (black ('0' in all color channels), white ('1' in all color channels), red, green, blue, magenta ('1' in red and blue channels), cyan ('1' in green and blue channels), and yellow ('1' in red and green channels)), suggesting synthetic artifacts. We provide visual examples (20 opposite images) in the following link:
https://drive.google.com/drive/folders/1Q85pz8y-36K2eHXDHsRcciblihvHgwgX?usp=drive_link

To quantify these differences, we compute Total Variation (TV), Entropy, and the percentage of extreme saturation values (top or bottom 1% of the pixel range). All metrics are computed per channel and averaged per image across three sets: original MSCOCO images, CLIP opposites, and W-CLIP opposites. The results are summarized below:

These findings confirm that W-CLIP opposites are statistically similar to natural images, whereas CLIP opposites exhibit significantly reduced entropy and variation and much higher percentage of saturation values, indicating a lack of natural structure.

Other Comments or Suggestions

"The figures in this paper are vague and have low resolution. For instance, subfigures in Figure 4 have small x-y axis labels and legends. Figure 3 is a good example, I highly recommend the authors redraw all the figures like Figure 3."

We thank the reviewer for this helpful suggestion. We will improve the visual quality of all figures in the final version. Specifically, we will enlarge the legends and the axes ticks in Figs. 4, 6, and 7, and reformat histograms for clearer presentation, following the style of Fig. 3.

We hope our additional experiments and clarifications address the concerns raised and further strengthen the validity and impact of our proposed approach. We would appreciate positively considering increasing the final rating of the paper.