Wasserstein Distortion: Unifying fidelity and realism

Thank you for the review. We are not aware of any background knowledge from image processing that is assumed in the paper. Could you mention some specific examples? Will we respond to your other comments later.

Thank you for your review. We agree that the contributions of the paper are not purely theoretical. We do not see this as a weakness, however. We see Wasserstein distortion as a generalization/synthesis of several lines of prior work on models of the human visual system, probabilistic texture models, and image realism/distortion.

The careful insights provided by the reviewers prompted us to replace the existing theorems (which are moved to the appendix) with a result and discussion on the metric properties of Wasserstein distortion and their connection to the spectral properties of the pooling PMF. In particular, we show that Wasserstein distortion is not a valid metric under popular choices of the pooling PMF due to the presence of spectral nulls. We feel that this provides a stronger theoretical justification for the approach and results in a stronger paper overall.

We believe our paper is self-contained, and could not identify any image processing concepts or terms that are relied upon in the paper that require domain knowledge.

Thank you for your thorough and incisive review. In response to the weaknesses that you highlighted, we updated our paper to now speak to the utility of nonuniform weights in several places:

We modified Section 3, and in the revised version we show that uniform weights do not result in a valid metric, because there are examples in which D(x,x’) = 0 even though x and x’ are different. Thus one cannot capture pure fidelity with this choice (note that most papers that use uniform weights are interested in realism, not fidelity). We show that the problem lies with the spectral properties of the uniform PMF. We added a new theorem, Theorem 3.1, in which we proved that this problem can be solved if one moves to nonuniform PMFs.

In experiment 1 in the current version (which was in Appendix B.2 in original submission), we find that nonuniform PMFs allow for a smooth transition from fidelity to realism, with an extensive intermediate regime. When using uniform PMFs, the transition is much more abrupt, to the extent that one does not obtain images that balance the two competing goals.

It is also worth noting that one cannot smoothly interpolate between fidelity and realism using uniform weights due to integer effects, which are especially pronounced in the regime of small pooling regions (such as when one moves from a region consisting of a single pixel to a region consisting of a 3x3 patch). Finally, we do not see the use of nonuniform weights as necessarily the primary contribution of the paper. Rather, we see our contribution as the identification of a distortion metric, in the mathematical sense, that subsumes fidelity and realism and allows for interpolation between them.

In Figure 8 in the original submission (Figure 13 in current version), we illustrated that a simple linear combination of pure fidelity and realism does not achieve the result we have; other simpler modifications, including [Ref1], would require more hand-tuning than we require: [Ref1] requires segmentation, and particular choices of guidance maps for fidelity and realism regions, and an indicator for each patch to be either fidelity or realism. Achieving smooth transitions would need yet more guidance maps. Our method only requires identifying the high fidelity region, which is either automatically calculated from the saliency map provided with the image, or automatically calculated once we know where the center of vision is. Also, we achieve a smoother transition than segmenting the image and processing each patch individually, as we show in the Figures 3,4,6,7 in the original paper (Figures 5,6,8,9 in the revised version). In comparison to other papers, multiple reference images are taken from standard references, Gatys et al. 2015, Ustyuzhaninov et al. 2017, etc. Comparing the results in these papers, we achieve comparable performance, even though our aim is not texture synthesis per se.

Some comparison of the computational requirements was mentioned in the discussion of Experiment 1 (now moved to Section B.1).

We are not sure what paper "Freeman et al. (2012)" refers to. Is the reviewer referring to J. Freeman and E. P. Simoncelli, “Metamers of the Ventral Stream,” Natural Neuroscience, vol. 14, pp. 1195-1201, 2011?

Thank you for your review. We would respond to the weaknesses you mention as follows: We assume the reference is to the paper: J. Freeman and E. P. Simoncelli, “Metamers of the Ventral Stream,” Natural Neuroscience, vol. 14, pp. 1195-1201, 2011.

This paper is indeed the point of departure for our work, as noted in the introduction. Our paper extends Freeman and Simoncelli in several ways:

Freeman and Simoncelli do not provide a distortion measure or loss function. They create perceptually indistinguishable images ("metamers") by constraining them to have the same local statistics across different patches. Thus the optimization problem they are solving has no objective; their goal is to find a feasible point, i.e., one that satisfies their constraints. Under their approach, whether two images are metamers is a binary notion; they do not quantify how far two images are from being metamers. Beyond being able to measure how close images are to being metamers (as in current Figure 12, Figure 10 in original submission), defining an explicit distortion measure enables the idea of measuring discrepancy between local statistics to be applied more broadly, such as in compression applications. It also allows one to precisely answer structural questions about the distortion measure, such as what properties of the pooling PMF we require, which brings us to the next point.

Section 3 of the paper has been replaced with a discussion of what properties we require in order for Wasserstein distortion to be a proper metric (in particular, to satisfy that d(x,x’) = 0 iff x = x’). We find that both the raised-cosine-type PMF used by Freeman and Simoncelli and the uniform PMF used in most other works is problematic in this regard. We explain why and suggest PMFs that are superior for our purposes.

Freeman and Simoncelli assume that the viewer focuses on the center of the image, and the pooling regions are the same for all images. We provide an automatic method for producing an image-specific sigma map from a given saliency map.

We connect the pooling idea of Freeman and Simoncelli to the recent literature on the distortion-perception tradeoff. In that literature, distortion and perception (or realism) are viewed as disparate constraints that are fundamentally different in character. The main goal of this paper is to show that they can be lifted in a common framework, inspired by Freeman and Simoncelli. The paper thus has the potential to change the way both the machine learning and information theory communities view the distortion-perception tradeoff.

We believe that we made the math in Section 2 as simple as possible and that it is inline with other ICLR papers. The introduction provides a non-mathematical description of the ideas as a warm-up, and the paper is not especially notation-heavy by the standards of the community. We define Wasserstein distortion and prove our theorems in 1-D for simplicity’s sake. Note that we do not make a global Gaussian assumption. In particular, Theorems 3.1 and 3.2 in the original submission (which are moved to Appendix C now) make no reference to the Gaussian distribution. Our experiments resort to means and variances for computational reasons, but one cannot pose Wasserstein distortion in terms of means and variances without compromising the generality of the theorems.

The sigma parameter does indeed correspond to the width of the pooling region in Freeman and Simoncelli, but we do not view this as a weakness of the paper.

While we would not describe Theorems 3.1 and 3.2 in the original submission as trivial (they require proofs, and the proofs are more than a few lines), they are intended to confirm that Wasserstein distortion has the extremal properties that one intuitively expects. We have moved these theorems to the appendix to avoid giving the impression that these results are difficult or unexpected. We replaced them with a result and discussion on how the metric properties of Wasserstein distortion relates to the spectral properties of the pooling PMF, which we believe will be of greater interest.

We are not sure what “vision study” refers to. The goal of the experiments was to validate the proposed distortion measure using the method propounded by Ding et al. (2021), namely by generating random images that are close to a given image under the proposed measure. In particular, we sought to illustrate how it can be useful to interpolate between fidelity and realism constraints. The goal was not to contribute to the literature on the human visual system. We have replaced Experiment 1 with a different experiment that interpolates between fidelity and realism without any reference to peripheral vision, which hopefully helps clarify this point.

Thank you for your careful and in-depth review. We especially appreciate your summary and description of the strengths of the paper. Regarding the weakness you mention, hallucination would appear to be intrinsic to any scheme that seeks a high level of realism without enforcing high pixel-level fidelity. Yet, image generation by minimization of Wasserstein distortion is less prone to hallucinating realistic elements that bear no resemblance to the original image because it relies entirely on local statistics of the image, without reference to the ensemble of all images. If the original image is, say, blurry, then we expect the reconstructed image to be blurry as well. In this sense the weakness you mention is less applicable to Wasserstein distortion than other methods of achieving realism, such as diffusion models. This was discussed in the last paragraph of the original submission. Also note that Wasserstein distortion, as a distortion measure derived from image statistics, is designed to quantify the extent to which a reproduced image contains realistic elements that are disconnected from the reference image. Rather than producing images that withstand close scrutiny in all regions, our image generation experiments are meant to demonstrate the properties of our metric, namely that it becomes increasingly permissive to error in the periphery, and that these errors are hard to spot when viewing the image at an appropriate distance.

We would answer your questions as follows:

In our experiments 2 and 3, the rate at which sigma grows as one moves away from the fovea is a parameter that we control. If we tune the parameter such that sigma grows slower, the reproduction would degenerate less. Thus the approach does not necessarily need to produce quickly degrading results. We do, in fact, iterate over the images in that we measure the Wasserstein distortion at a subset of points, for computational reasons, and this subset is randomly selected from iteration to iteration. The lack of consistency with respect to lines seems to be an artifact of our use of VGG19. Other texture generators that use VGG19 have similar limitations, and various papers have proposed ways of augmenting VGG19 to address this (see the references in the caption for Figure 7 in the original submission which is Figure 9 in the revision). This theory is supported by the final row of Figure 9, which is new in the revision. This is simply a repeat of the third row but with a downsampled image. We see that the foul lines are much more consistent, presumably because VGG-19 is better able to capture long-range dependence on the downsampled image. We emphasize that our method is agnostic to the choice of features and the various improvements to VGG-19 cited in the caption to Figure 9 could in principle be incorporated into our framework.

Wasserstein distortion cannot detect generated or edited content because it is a full-reference distortion measure. Thus it can only measure the distance between two images. It cannot score the likelihood or realism of a single image in isolation.

For the most complicated experiment we have, namely experiment 3, it took about 3 hours per image (which are 480x480) on an NVidia GTX4090 to go from a completely random image to the final image shown in the paper. We added this information in Appendix B in our revised submission. Some discussion of complexity is included in Appendix B.1 (which was included in the paper itself in the original submission).