The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric

We thank the reviewer for the questions regarding ablations and interest in the correlation between perceptual and downstream tasks, of which answers will be incorporated into the main text.

We hope our rebuttal addresses your concerns and you will consider increasing your score.

The experimental dataset consists only BAPPS.

We are aware of the existence of other full-reference (i.e. pairwise) IQA datasets, however BAPPS is the only one we know of that contains a rich set of distortions. Other established datasets such as TID2013 or LIVE do not contain CNN-based distortions such as warping, and as such are prone to overfitting to (which we believe has already happened with traditional metrics such as MS-SSIM). A topic for future work would be to collect our own data to validate on, but this is unfortunately outside of the scope of this paper.

Perhaps LASI can only measure perceptual distance at patch level, and will it be useful for high-resolution images?

Distortions are usually applied to the image as a whole and therefore different patches are distorted in a similar way. It is common in the FR-IQA literature to scale up to larger images by computing the metric on smaller patches for this reason. LASI can be run in the same way by parallelizing over patches and averaging the distance, a common technique in the literature. We expect this to perform similarly, the same way other methods do, and will include an experiment on LIVE IQA [1] in the follow-up.

[1] Sheikh H. LIVE image quality assessment database release 2. http://live.ece.utexas.edu/research/quality. 2005.

An ablation study is missing for the causal neighborhood and non-causal neighborhood, as semantic understanding relies on contextual relationships around pixels or regions.

We thank the reviewer for this suggestion of what is effectively a new version of our LASI metric. This exact ablation has been run and is shown below. The performance of the new suggestion is slightly worse, however, the implementation is simplified as computing the non-causal mask consumes less memory.

Please click for ablation plot

The sentence in the section 4.1 ‘’Our method relies on self-supervision (at inference-time) to learn a representation for each pixel that captures global perceptual semantics of the image. The underlying assumption is that, given a representation vector for some pixel, it must successfully predict the value of other pixels in the image in order to capture useful semantics of the image’s structure.” But the relationship of the perceptual embedding in FR-IQA and the semantic extraction is confused in this work. Because many previous methods also use semantic features extracted by pre-trained models on high-level classification tasks to calculate perceptual distance. Is there any difference between this semantic feature and the embedding derived by Eq 1 in this work?

We are not sure if we understand your question correctly, but we do not think that we are confusing two different concepts in this work. Computing a perceptual embedding of an image in many prior works is done via a function $f(x; \theta)$, where $x$ is the image and $\theta$ are, for example, neural network parameters that are determined by a separate training stage. In LASI, computing the weights is done via a function $f(x)$ that does not have trainable parameters, hence no training stage is necessary. Computing the embedding is performed via solving a linear least squares problem, and hence there is no analytic formula for $f$. However, that does not matter, since $f$ is still a deterministic function of the image: every time we solve for $w$ (the weights of the linear least squares problem), we get the same answer.

Since the LASI is designed on the semantic extraction in the images’ structure, so the correlation of the prediction task and perceptual task is good, which is a little obvious.

There is no evidence in the psychophysical literature that points towards the human visual system performing low-level reconstruction tasks. It is not immediately obvious to us that good perceptual embeddings can be learned from linear prediction of pixels.

However, we agree that it is intuitive to expect the correlation to be better for LASI (prediction) than for LPIPS (classification). The reason being that a useful feature for classification is not necessarily useful for visual tasks regarding the image’s structure.

We would like to thank the reviewer for their questions regarding compute time which we will incorporate the response into the main text. LASI (ours) requires less compute than DNN methods due to their expensive forward-pass.

We hope our rebuttal addresses your concerns and you will consider increasing your score.

The paper claims that the advantage of the proposed method is that it requires no training data or DNN features. But, the method requires computations at inference time. Considering that once training is done, DNN-feature based methods do not require much of extra computations (hence the cost is amortized) while the proposed method requires extra computations, is requiring training data or DNN features really a bad thing?

Even with amortization, computing DNN features requires more computation at inference time than LASI (ours). DNN features require a forward-pass in the neural network to compute embeddings. The plot below shows the wall-time for computing PIM (a DNN) and LASI (ours) distances, compiled with tf.function and jax.jit, respectively. Note how LASI is faster for the neighborhood sizes considered in the experiments (i.e., $10$ to $12$).

Please click for LASI and PIM comparison plot

The paper claims to have competitive performance, compared with LPIPS method. But, 11% difference in SR in Table 1 seems rather high.

The difference mentioned by the reviewer is with respect to supervised LPIPS, i.e., after fine-tuning LPIPS on human annotated data. When comparing to unsupervised LPIPS (i.e., neural network embeddings before fine-tuning), which we hold to be the fairer comparison, the difference is less than $2\\%$.

We will call attention to this fact in the caption of Table 1.

All experiments are performed with 64x64 resolution, which seem rather small. Is the proposed method effective for larger images, compared to other works?

Distortions are usually applied to the image as a whole and therefore different patches are distorted in a similar way. It is common in the FR-IQA literature to scale up to larger images by computing the metric on smaller patches for this reason. LASI can be run in the same way by parallelizing over patches and averaging the distance, a common technique in the literature. We expect this to perform similarly, the same way other methods do, and will include an experiment on LIVE IQA [1] in the follow-up.

[1] Sheikh H. LIVE image quality assessment database release 2. http://live.ece.utexas.edu/research/quality. 2005.

Is the choice of $N$ the size of neighborhood robust against the image size? It seems $N$ needs to be tuned for each image resolution, which can be critical, since images can come in at various sizes.

$N$ does not need to be tuned for each image resolution. Our experiments show increasing $N$ gives better downstream performance (see Figure 3). $N$ should be set to as large as possible while respecting the compute and memory budget of the application.

The paper claims that the method can be combined with LPIPS but I cannot find the experimental results on this. Does the combination actually bring improvements?

The MAD experiment highlights that LPIPS and LASI fail in different ways. We experimented with merging these models by taking geometric averages of their distances. This resulted in a slightly better perceptual performance but not enough to justify merging, as the compute required is now that of LPIPS + LASI.

Merging these models is a non-trivial manner as the embedding dimensions differ in size making it difficult to perform simple aggregation techniques at the embedding level. Note this is true in general for any pair of models.

We will update the writing of the paper to make this clear in the main text.

Thanks for the efforts and clarifications that have addressed my concerns. Thus, I have updated the score accordingly.

We thank the reviewer for the questions regarding the surprising performance of LASI (ours) over neural network baselines. Indeed the performance of LASI (ours) is extremely close to that of LPIPS and PIM, which is the central point of this paper.

We hope our rebuttal addresses your concerns and you will consider increasing your score.

There is minimal discussion about the failure cases using the proposed method. Would be great to have some qualitative results and the probable reason we are seeing the results as we see it.

Authors fails to discuss adequately why in some cases other metrics (such as PIM) excel compared to the LASI metric.

As mentioned in the weakness section, it is imperative that authors present more details qualitative and quantitative about the failure cases seen using the LASI method proposed in the paper.

The difference in performance between LASI (ours) and PIM is insignificant (less than $2.3\\%$ averaged across categories) leaving very little signal to perform a detailed analysis, which is the main point of the paper.

A qualitative analysis on the examples of the dataset yields no immediately obvious pattern that would explain this small difference in performance.

The comparison in Table 1 favors alternative metrics by design. LASI (ours) is held fixed while the best performing models for other metrics are chosen for each category. For example, for LPIPS we picked the best performing embedding between VGG, SqueezeNet, and AlexNet, for each category, while for PIM we chose between PIM-1 and PIM-5 (whichever scored higher). This cherry picking explains the gap: if we tune LASI (ours) by varying $\omega$, then the gap can be removed.

This is exactly the main message of the paper: we can achieve perceptual performance virtually indistinguishable from that of neural network methods while using no training data or deep features, even when neural network methods are allowed to be fine-tuned for each category.

In the results presented in Table:1, why does MS-SSIM outdo the author's proposed method for the BAPPS-JND task.. whereas it outperforms it for the BAPPS-2AFC task?

MS-SSIM does not outdo our method on the JND task. This is only true in one category of the JND task and by less than $1\\%$, while in the other category our method is superior by $54.4\\%$.

In all other categories, for both 2AFC and JND, LASI (ours) outperforms MS-SSIM.

This is highlighted in Table 1, in the last row labeled "Improv. over MS-SSIM", where only the last column is negative ($-0.9\\%$) and all others are positive.

In Section 5.3: "....Results indicate LASI can find failure points for LPIPS and vice-versa......" Can authors elaborate on this point ?

Images in the box labeled “LPIPS loss fixed” have the same LPIPS distance to the reference. This means LPIPS says all images have equal perceptual quality relative to the reference. However, clearly the bottom row has better perceptual quality than the top row. This is known as a “failure mode” in the MAD literature.

Conceptually, this is analogous to finding adversarial examples for a classifier – all known perceptual metrics have failure modes. In conducting this experiment, we were hoping to find qualitative explanations for the unexpected performance of LASI, but the results have so far been inconclusive.