Idempotence and Perceptual Image Compression

• W4. For BD-PSNR, we emphasis that:

  • We can control our decoder to achieve an intermediate point between FID and MSE. In Appendix B.3 and Fig. 8, we can see that by trading FID for MSE, our approach achieves better FID with comparable MSE with ILLM. And it can also achieves comparable FID with better MSE compared with ILLM. We also note that it is impossible to achieve sota FID and sota PSNR at the same time by distortion-perception trade-off [Blau 2018]. Therefore, we can not achieve a PSNR as good as [Balle 2018] and [He 2022].
  • Our MSE is within 2x MSE of base codec, and our PSNR is within 3 dB of base codec, which is aligned to the theoretical result.

• For other metrics, we have evaluated our approach additionally on KID and LPIPS:

  Method	FFHQ		ImageNet		COCO		CLIC
  	BD-logKID↓	BD-LPIPS↓	BD-logKID↓	BD-LPIPS↓	BD-logKID↓	BD-LPIPS↓	BD-logKID↓	BD-LPIPS↓
  MSE Baselines
  Hyper	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
  ELIC	-0.2320	-0.0402	-0.348	-0.0580	-0.236	-0.0623	-0.4067	-0.0595
  BPG	0.1506	-0.0102	0.0274	-0.0102	0.1267	-0.0116	0.03985	-0.00795
  VTM	-0.2320	-0.0311	-0.2975	-0.0476	-0.2157	-0.0504	-2.049	-0.04812
  Conditional-based
  HiFiC	-3.132	-0.1087	-2.274	-0.1724	-2.049	-0.1723	-1.925	-0.1483
  HiFiC*	-4.261	-0.1102	-2.780	-0.1734	N/A	N/A	N/A	N/A
  Po-ELIC	-3.504	-0.1047	-2.877	-0.1674	-2.671	-0.1687	-2.609	-0.145
  CDC	-2.072	-0.0600	-1.968	-0.0985	-1.978	-0.1011	-2.122	-0.08437
  ILLM	-3.418	-0.1092	-2.681	-0.1809	-2.62	-0.1805	-2.882	-0.1547
  ILLM*	-4.256	-0.1062	-2.673	-0.1784	N/A	N/A	N/A	N/A
  Unconditional-based
  Proposed (Hyper)	-5.107	-0.0859	-4.271	-0.0576	-4.519	-0.0826	-3.787	-0.0557
  Proposed (ELIC)	-5.471	-0.0987	-5.694	-0.1062	-5.36	-0.113	-4.046	-0.0796

• The result of KID has same trend as FID, which means that our approach is sota. While many other approaches outperform our approach in LPIPS. This result is expected, as:

  • KID is a divergence based metric. And our approach is optimized for divergence.
  • LPIPS is a image-to-image distortion. By distortion-perception trade-off [Blau 2018], perception optimized approaches can not achieve SOTA LPIPS.
  • All other perceptual codec (HiFiC, ILLM, CDC, Po-ELIC) except for ours use LPIPS as loss function during training.
  • ILLM [Muckley 2023] also achieves sota FID, KID and visual quality at that time, but its LPIPS is outperformed by a autoencoder trained with LPIPS.

Thanks for your detailed review. And we are glad to provide our answer to your questions:

• W1. Although we have provided proof sketch in main text and complete proof in Appendix of Theorem 1 and Theorem 2, we do agree that they are quite abstract and not intuitive. However, theoretical results take $50$ percent of this paper. And it is a must to understand them correctly to understand this paper. To provide a more intuitive illustration, we will include an example with discrete finite alphabet (our theoretical results are appliable to any countable alphabet or Borel-measurable source $X$ and any deterministic measureable function $f(.)$):
  • Example 1 (1-dimension, discrete finite alphabet, optimal 1-bit codec): Consider discrete 1d random variable $X$, with alphabet $\mathcal{X} = \{0,1,2,3,4,5\}$, and $Y$ with alphabet $\mathcal{Y}=\{0,1\}$. We assume $P(X=i) = \frac{1}{6}$, i.e., $X$ follows uniform distribution. We consider a deterministic transform $f(X): \mathcal{X}\rightarrow \mathcal{Y} = \textrm{round}(X/3)$. Or to say, $f(.)$ maps $\{0,1,2\}$ to $\{0\}$, and $\{3,4,5\}$ to $\{1\}$. This is infact the MSE-optimal 1-bit codec for source $X$ (See Chapter 10 of [Elements of Information Theory]). And the joint distribution $P(X,Y)$, posterior $P(X|Y)$ is just a tabular

  • x	y	P(X=x,Y=y)	P(X=x|Y=y)	f(X=x)
    0	0	1/6	1	0
    1	0	1/6	1	0
    2	0	1/6	1	0
    3	0	0	0	1
    4	0	0	0	1
    5	0	0	0	1
    0	1	0	0	0
    1	1	0	0	0
    2	1	0	0	0
    3	1	1/6	1	1
    4	1	1/6	1	1
    5	1	1/6	1	1

  • We first examine Theorem 1, which says that any sample from $X\sim P(X|Y=y)$ satisfies $f(X) = y$. Observing this table, this is indeed true. As for $f(X)\neq y$, the posterior $P(X=x|Y=y)$ is $0$.
  • We then examine Theorem 2, which says that sampling from $X\sim P(X)$ with $f(X)=y$ constraint is the same as sampling from $P(X|Y=y)$. We first identify the set such that $f(X)=y$. When $y=0$, this set is $\{0,1,2\}$. And when $y=1$, this set is $\{3,4,5\}$. And sampling from $p(X)$ with constrain $f(X)=y$ is equvalient to sampling from one of the two subset with probability $\propto p(X)$. And this probability is exactly $P(X|Y=y)$.
• W2. The "Idempotence brings perceptual quality" in Sec. 3 is Theorem 2. It is already theoretically supported by proof in Appendix. A. Furthermore, it is already empirically supported by the Sec. 5. In Tab. 2 and Fig. 5 of Sec. 5, we can clearly see that indeed optimizing idempotence loss brings perceptual image compression.
• W3. It could be the problem of PDF compression. We have provided a version with larger resolution in https://ibb.co/NWww3RX. Probably zoom in can help (Reviewer VEis already finds qualitative results compelling with the PDF). We note that only our approach can reconstruct the beard of dog, while HiFiC and ILLM fail to do so. Further, our reconstruction has much more realistic texture of dog's hair.

Sure, below we present the result of MS-SSIM:

In terms of MS-SSIM, ELIC > HiFiC > ILLM > Hyper > Proposed (ELIC) > Proposed (Hyper). We are reluctant to use MS-SSIM as perceptual metric, as this result is obviously not aligned with visual quality, and should not be used when we considering divergence based perceptual quality as [Blau 2018], because:

• MS-SSIM is an image to image distortion. By [Blau 2018], it is in odd with divergence based metrics such as FID, KID. Theoretically, there does not exist a codec that achieve optimal FID and MS-SSIM at the same time.
• In terms of MS-SSIM, ELIC, a mse optimzied codec, is the sota. This indicates that MS-SSIM correlates poorly with human perceptual quality.
• Similar result is also reported in [Muckley 2023] Fig. 3, where the MS-SSIM of ILLM is not even as good as Hyper, which is a mse optimized codec. This indicates that MS-SSIM correlates poorly with human perceptual quality.
• In a perceptual codec competition CVPR CLIC 2021 (https://clic.compression.cc/2021/leaderboard/image-075/test/index.html), the human perceptual quality is tested as final result and MS-SSIM, FID are evaluated. The 1st place of human perceptual test result among 23 teams "MIATL_NG", also has lowest FID, which indicates FID correlates well with human perceptual quality. On the other hand, its MS-SSIM ranks 21st place among 23 teams, which indicates MS-SSIM correlates poorly with human perceptual quality.

We will include MS-SSIM result in our final version, but we emphasis that this result should be treated with care.

Thank the authors for the reply. I partly insist on my point that I can't see the superiority of the proposed method in the visual comparison with HiFiC and ILLM without the comparison of MS-SSIM. If the authors want to change my point, please show those real results to me. Even if the results are not better than HiFIC and ILLM, I will increase my score.

• Q1. Thanks for the advice, we will add this discussion. Our theory relies on the assumption that the generative model perfectly learns the natural image distribution, while this is hard to achieve in practice. And we think this is where the gap lies. More specifically, GAN has better "precision" and worse "recall", while diffusion has a fair "precision" and fair "recall". The "precision" and "recall" is defined in [Improved precision and recall metric for assessing generative models]. Or to say, the distribution learned by GAN largely lies in the natural image distribution better, but many area of natural image distribution is not covered. The distribution learned by diffusion does not lies in natural image distribution as good as GAN, but it covers more area of natural image distribution. This phenomena is also observed and discussed by [Diffusion Models Beat GANs on Image Synthesis] and [Classifier-Free Diffusion Guidance]. The "precision" is more important for sampling, but "recall" is more important for inversion. That is why the GAN based approach fails. The MCG fails probably because it is not well suited for non-linear problems.
• Q2. [Agustsson 2022] [Muckley 2023] evaluate FID with pathsize 256 on 256x256 COCO dataset with 30000 images and around 2000x2000 CLIC dataset with 400 images. The former produces 30000 patches and the later produces around 15000 pathes. As our approach is more expensive to evaluate, we can not afford testing on that large dataset. Therefore, we test our approach on 256x256 COCO dataset with 1000 images and 256x256 CLIC dataset with 400 images. In that case, using FID/256 will produce only 1000 and 400 patches. As FID is biased [Effectively Unbiased FID and Inception Score and where to find them], 1000 datapoints is too small for FID evaluation. Therefore, we evaluate FID with 64x64 patches, which produce 16000 and 6400 datapoints, which is reasonablely large. And this patch number is close to [Agustsson 2022] [Muckley 2023]. Note FID is proposed to be used with 64x64 CelebA and 64x64 ImageNet [Two time-scale update rule for training GANs]. So we think it is fine to use FID with 64x64 patches.
• Q3. Sure, we can draw the perception-distortion trade-off as Fig. 1 of [Muckley 2023], and the result is shown in https://ibb.co/DrtyXTG. We choose this way because we can compare to other methods easily, while the way of [Muckley 2023] can only compare to ourselves as we can not match the bitrate of other methods without selecting the bitrate of our base model carefully.
• Q4. Thanks for this advice, we will revise it into divergence-based perception to distinguish it with human perception.

• W4. Sure, we have evaluated our approach additionally on KID and LPIPS:

  Method	FFHQ		ImageNet		COCO		CLIC
  	BD-logKID↓	BD-LPIPS↓	BD-logKID↓	BD-LPIPS↓	BD-logKID↓	BD-LPIPS↓	BD-logKID↓	BD-LPIPS↓
  MSE Baselines
  Hyper	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
  ELIC	-0.2320	-0.0402	-0.348	-0.0580	-0.236	-0.0623	-0.4067	-0.0595
  BPG	0.1506	-0.0102	0.0274	-0.0102	0.1267	-0.0116	0.03985	-0.00795
  VTM	-0.2320	-0.0311	-0.2975	-0.0476	-0.2157	-0.0504	-2.049	-0.04812
  Conditional-based
  HiFiC	-3.132	-0.1087	-2.274	-0.1724	-2.049	-0.1723	-1.925	-0.1483
  HiFiC*	-4.261	-0.1102	-2.780	-0.1734	N/A	N/A	N/A	N/A
  Po-ELIC	-3.504	-0.1047	-2.877	-0.1674	-2.671	-0.1687	-2.609	-0.145
  CDC	-2.072	-0.0600	-1.968	-0.0985	-1.978	-0.1011	-2.122	-0.08437
  ILLM	-3.418	-0.1092	-2.681	-0.1809	-2.62	-0.1805	-2.882	-0.1547
  ILLM*	-4.256	-0.1062	-2.673	-0.1784	N/A	N/A	N/A	N/A
  Unconditional-based
  Proposed (Hyper)	-5.107	-0.0859	-4.271	-0.0576	-4.519	-0.0826	-3.787	-0.0557
  Proposed (ELIC)	-5.471	-0.0987	-5.694	-0.1062	-5.36	-0.113	-4.046	-0.0796

• The result of KID has same trend as FID, which means that our approach is sota. While many other approaches outperform our approach in LPIPS. This result is expected, as:

  • KID is a divergence based metric. And our approach is optimized for divergence.
  • LPIPS is a image-to-image distortion. By distortion-perception trade-off [Blau 2018], perception optimized approaches can not achieve SOTA LPIPS.
  • All other perceptual codec (HiFiC, ILLM, CDC, Po-ELIC) except for ours use LPIPS as loss function during training.
  • ILLM [Muckley 2023] also achieves sota FID, KID and visual quality at that time, but its LPIPS is outperformed by a autoencoder trained with LPIPS.

Thanks for your detailed review. And we are glad to provide our answer to your questions:

• W1. The reason why we only compare to CDC among diffusion codec is that it is the only diffusion codec that provides test code. Furthermore, [Goose 2023] [Hoogeboom 2023] are not yet published. Not comparing to them should not be considered as a weakness. Our testing dataset has different size and resolution than [Agustsson 2022], and that is why we have not compared to the image released. However, we do note that [Muckley 2023] outperform the datapoint released by [Agustsson 2022] in terms of FID, and we outperform [Muckley 2023] in terms of FID.

• W2. We are not sure if we get this correctly, so please correct us if our understanding is wrong. It appears to us that [Hoogeboom 2023] is a conditional generative codec, while its condition is MSE reconstruction $g_0(f_0(x))$. The diffusion part takes $g_0(f_0(x))$ as condition, and rectified flow allows very fast sampling within a few steps. We search for the word "unconditional" in [Hoogeboom 2023] but we fail to find relevant information. Thus it appears to me that [Hoogeboom 2023] is a conditional codec using $g_0(f_0(x))$ as condition, while x-domain constraint is $||g_0(f_0(\hat{x}))-g_0(f_0(x))||$. They both contains $g_0(f_0(x))$, but we do not think they are identical.

• W3. Sure, we re-evaluate the original tensorflow HiFiC, and the results are as follows:

  	FFHQ		ImageNet		COCO		CLIC
  	BD-FID↓	BD-PSNR↓	BD-FID↓	BD-PSNR↓	BD-FID↓	BD-PSNR↓	BD-FID↓	BD-PSNR↓
  HiFiC (Pytorch)	-48.35	-2.036	-44.52	-1.418	-44.88	-1.276	-36.16	-1.621
  HiFiC (Tensorflow)	-46.44	-1.195	-40.25	-0.8287	-46.45	-0.9176	-35.90	-0.9202

• In terms of BD-FID, the torch version outperforms tf version in FFHQ, ImageNet and CLIC dataset, while it is outperformed by tf version in COCO. The overall conclusion is not impacted.

Thanks for your detailed review. And we are glad to provide our answer to your questions:

• W1. As we have discussed in limitation, the speed is an issue shared by all diffusion-inversion approaches. Currently, the latest preprint in this area [Prompt-tuning latent diffusion models for inverse problems] using fast samplers, such as DDIM [Denoising Diffusion Implicit Models], still need the same 1000 step as DDPM. Even though DDIM works well with 50 steps for non-inverse problems. We have also attempted DDIM with fewer steps by ourselves, but we can not make it work. Thus, simply chaning DDPM to faster sampler does not work. The diffusion-inversion community is very active and we believe that the situation can be improved soon.
• W2. As we have discussed in limitation, we agree that the resolution is a problem. We have done some very early attempt using diffusion model with larger resolution. For example, our approach works with 512x512 Stable Diffusion 1.5 and null text prompt. And a visual example with [Balle 2018] and ImageNet 512x512 Image is: https://ibb.co/xCd9gMB.
• We have not tried 1024x1024 Stable Diffusion 2.0. Note that as Stable Diffusion is a latent diffusion, the latent space size is much smaller than pixel size. And thus its complexity does not grow that much. Currently our higher-resolution codec relies on higher-resolution diffusion model. We are also aware that there are other pathch-wise solutions [DiffCollage: Parallel Generation of Large Content with Diffusion Models], but they are not the mainstream in diffusion community and we have not given them a try.
• For the baselines. we note that all the baselines are trained on 256x256 images. And to secure fair comparsion, we also provide the result of HiFiC and ILLM re-trained on 256x256 FFHQ and ImageNet in Tab.2.
• Q1. The weird reconstruction is due to an improper hyper-parameter. More specifically, the scale parameter $\zeta=0.3$ is too small, which lead to the reconstruction deviate from original image. We have shown additional results varying $\zeta=0.6$ in https://ibb.co/XXRQN2F. Clearly use $\zeta=0.6$ improve the result.

• Q1. By [Blau 2018] and Corollary 1, the MSE of inversion-based perceptual codec is bounded by 2x base codec. Using a perceptual codec as base codec:

  • does not help with perceptual quality, as theoretically any base codec inversion achieves perfect perceptual quality.
  • does loosen MSE upperbound, as the current upperbound is 2x MSE of a perceptual codec. And it is very likely that the MSE of our codec grows larger.

• Therefore, we choose to use a MSE optimized codec, to obtain the tightest MSE bound possible.

• Q2. Sure, we have evaluated our approach additionally on KID and LPIPS:

  Method	FFHQ		ImageNet		COCO		CLIC
  	BD-logKID↓	BD-LPIPS↓	BD-logKID↓	BD-LPIPS↓	BD-logKID↓	BD-LPIPS↓	BD-logKID↓	BD-LPIPS↓
  MSE Baselines
  Hyper	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
  ELIC	-0.2320	-0.0402	-0.348	-0.0580	-0.236	-0.0623	-0.4067	-0.0595
  BPG	0.1506	-0.0102	0.0274	-0.0102	0.1267	-0.0116	0.03985	-0.00795
  VTM	-0.2320	-0.0311	-0.2975	-0.0476	-0.2157	-0.0504	-2.049	-0.04812
  Conditional-based
  HiFiC	-3.132	-0.1087	-2.274	-0.1724	-2.049	-0.1723	-1.925	-0.1483
  HiFiC*	-4.261	-0.1102	-2.780	-0.1734	N/A	N/A	N/A	N/A
  Po-ELIC	-3.504	-0.1047	-2.877	-0.1674	-2.671	-0.1687	-2.609	-0.145
  CDC	-2.072	-0.0600	-1.968	-0.0985	-1.978	-0.1011	-2.122	-0.08437
  ILLM	-3.418	-0.1092	-2.681	-0.1809	-2.62	-0.1805	-2.882	-0.1547
  ILLM*	-4.256	-0.1062	-2.673	-0.1784	N/A	N/A	N/A	N/A
  Unconditional-based
  Proposed (Hyper)	-5.107	-0.0859	-4.271	-0.0576	-4.519	-0.0826	-3.787	-0.0557
  Proposed (ELIC)	-5.471	-0.0987	-5.694	-0.1062	-5.36	-0.113	-4.046	-0.0796

• The result of KID has same trend as FID, which means that our approach is sota. While many other approaches outperform our approach in LPIPS. This result is expected, as:

  • KID is a divergence based metric. And our approach is optimized for divergence.
  • LPIPS is a image-to-image distortion. By distortion-perception trade-off [Blau 2018], perception optimized approaches can not achieve SOTA LPIPS.
  • All other perceptual codec (HiFiC, ILLM, CDC, Po-ELIC) except for ours use LPIPS as loss function during training.
  • ILLM [Muckley 2023] also achieves sota FID, KID and visual quality at that time, but its LPIPS is outperformed by a autoencoder trained with LPIPS.

Thanks for your detailed review. And we are glad to provide our answer to your questions:

• W1. First, migrating from super-resolution to codec is not that straightforward. Unlike super-resolution and other restortation, codec is highly non-linear. And many previously proposed approaches that work well for super-resolution, such as [Menon 2020] [Daras 2021] [Chung 2022a], fail. As we have shown in Tab. 1, a thorough curation of different generative model and inversion approach is required to make it actually work. Furthermore, in Sec. 4.2, we additionally propose to use the straight through estimation to make codec differentiable, and the x-domain y-domain constraint. Those techniques are codec specific and have never been studied in super-resolution.
• Second, indeed we borrow the empirical approach from super-resolution. However, we also give back theoretical understanding. Despite inversion-based super-resolution has been studied for years empirically, their theoretical relationship with conditional model based super-resolution, and distortion-perception trade-off is in general unknown. PULSE [Menon 2020] is the poineer of this area, and they justify their approach by "natural image manifold" assumption. And later works in inversion-based super-resolution follow PULSE's story. On the other hand, our Theorem 1, Theorem 2 only limit the operator $f(.)$ to be deterministic, and it need not to be a codec. Then, our Theorem 1, Theorem 2 and Corollary 1 are also appliable to inversion based super-resolution, at least when the degeneration operator is deterministic. In other words, our theoretical results also implies that:
  • Theorem 1' (super-resolution version) Conditional generative super-resolution also satisfies idempotence, that is, the up-sampled image can downsample into low-resolution image.
  • Theorem 2' (super-resolution version) Inversion-based super-resolution is theoretically equvalient to conditional generative super-resolution.
  • Corollary 1' (super-resolution version) Inversion-based super-resolution satisfies the theoretical results of distortion-perception trade-off, that is the MSE is at most double of best MSE.
• We believe that our result provides non-trivial theoretical insights to inversion-based super-resolution community. For example, [Menon 2020] claim that the advantage of inversion-based super-resolution over the conditional generative super-resolution is that inversion-based super-resolution "downscale correctly". However, with our theoretical result, we know that ideal conditional generative super-resolution also "downscale correctly". Currently they fail to achieve this due to implemention issue. Another example is that most inversion-based super-resolution [Menon 2020] [Daras 2021] [Chung 2022a] report no MSE comparsion with sota MSE super-resolution, as it is for sure that their MSE is worse and there seems to be no relationship between their MSE and sota MSE. However, with our theoretical result, we know that their MSE should be smaller than 2x sota MSE. And they should examine whether their MSE falls below 2x sota MSE.